-- Logs begin at Sat 2022-05-07 20:01:27 UTC, end at Sat 2022-05-07 23:01:54 UTC. -- May 07 21:36:31.949801 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. May 07 21:36:35.360723 ncdv-master-1 cinder-backup[129416]: INFO cinder.cmd.backup [-] Backup running with 1 processes. May 07 21:36:35.361611 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=129416) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} May 07 21:36:35.362127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=129416) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} May 07 21:36:35.537564 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_db.sqlalchemy.engines [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129416) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 21:36:35.779080 ncdv-master-1 cinder-backup[129416]: INFO cinder.rpc [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. May 07 21:36:35.783306 ncdv-master-1 cinder-backup[129416]: INFO cinder.rpc [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. May 07 21:36:35.791919 ncdv-master-1 cinder-backup[129416]: INFO cinder.rpc [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. May 07 21:36:35.796318 ncdv-master-1 cinder-backup[129416]: INFO cinder.rpc [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. May 07 21:36:35.801328 ncdv-master-1 cinder-backup[129416]: INFO cinder.rpc [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. May 07 21:36:35.805936 ncdv-master-1 cinder-backup[129416]: INFO cinder.rpc [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. May 07 21:36:35.860458 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_db.sqlalchemy.engines [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129416) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 21:36:35.882709 ncdv-master-1 cinder-backup[129416]: INFO oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Starting 1 workers May 07 21:36:35.888607 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Started child 129863 {{(pid=129416) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} May 07 21:36:35.889632 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Full set of CONF: {{(pid=129416) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} May 07 21:36:35.890027 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] ******************************************************************************** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} May 07 21:36:35.890356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] Configuration options gathered from: {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} May 07 21:36:35.890876 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} May 07 21:36:35.891650 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} May 07 21:36:35.892149 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] ================================================================================ {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] allow_availability_zone_fallback = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] allow_compression_on_image_upload = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] allowed_direct_url_schemes = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] api_rate_limit = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] auth_strategy = keystone {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_availability_zone = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.901977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_api_class = cinder.backup.api.API {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_chunk_size = 134217728 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_image_journals = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_pool = backups {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_stripe_count = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_stripe_unit = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.902645 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_ceph_user = cinder-bak {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903212 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903212 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_driver_init_check_interval = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903212 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_driver_stats_polling_interval = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903212 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903212 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_max_operations = 15 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903212 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_metadata_version = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903683 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_name_template = backup-%s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903683 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_native_threads_pool_size = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.903683 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_object_number_per_notification = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.904153 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_service_inithost_offload = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.904517 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_timer_interval = 120 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.904984 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_use_same_host = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.905476 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_use_temp_snapshot = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.905955 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backup_workers = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.906511 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] chap_password = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.906963 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] chap_username = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.907426 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.907849 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_internal_tenant_project_id = 758d4dea899344b883971712aebe042f {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.908100 ncdv-master-1 cinder-backup[129863]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) May 07 21:36:35.908619 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_internal_tenant_user_id = cf4eae7f407e44ecbe525049b262e3a5 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.908619 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] compression_format = gzip {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.909080 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] config_dir = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.909387 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.909787 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] config_source = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.910150 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.910611 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] control_exchange = openstack {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.911095 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] debug = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.918814 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] default_availability_zone = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.918814 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] default_group_type = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.918814 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.918814 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] default_volume_type = ceph {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.918814 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] driver_client_cert = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] driver_client_cert_key = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] driver_data_namespace = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] driver_ssl_cert_path = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] driver_ssl_cert_verify = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] driver_use_ssl = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] enable_new_services = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919322 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] enable_unsupported_driver = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] enabled_backends = ['ceph'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] enforce_multipath_for_image_xfer = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] filter_function = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_api_insecure = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_api_servers = ['http://10.222.0.32/image'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.919868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_api_ssl_compression = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.920348 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_ca_certificates_file = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.920619 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_catalog_info = image:glance:publicURL {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.921082 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.921685 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.922080 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.922566 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_num_retries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.922943 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] glance_request_timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.923310 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] goodness_function = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.923733 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] graceful_shutdown_timeout = 5 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.924117 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] group_api_class = cinder.group.api.API {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.924482 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] host = ncdv-master-1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.924885 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iet_conf = /etc/iet/ietd.conf {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.925347 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_compress_on_upload = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.925708 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_conversion_address_space_limit = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.926194 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_conversion_cpu_limit = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932121 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932121 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_upload_use_cinder_backend = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932121 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_upload_use_internal_tenant = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932121 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_volume_cache_enabled = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932121 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_volume_cache_max_count = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932121 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] image_volume_cache_max_size_gb = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] instance_format = [instance: %(uuid)s] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iscsi_iotype = fileio {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iscsi_secondary_ip_addresses = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iscsi_target_flags = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iscsi_write_cache = on {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.932834 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iser_helper = tgtadm {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.933348 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iser_ip_address = 10.222.0.32 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.933348 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iser_port = 3260 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.933532 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.934126 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_config_append = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.934513 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.935089 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_dir = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.935518 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_file = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.935868 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_options = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.936380 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_rotate_interval = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.936898 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_rotate_interval_type = days {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.937287 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] log_rotation_type = none {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.937688 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.937994 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.943796 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.943796 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.943796 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.943796 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] max_age = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.943796 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] max_logfile_count = 30 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.943796 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] max_logfile_size_mb = 200 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] max_over_subscription_ratio = 20.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] message_reap_interval = 86400 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] message_ttl = 2592000 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] monkey_patch = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] monkey_patch_modules = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] my_ip = 10.222.0.32 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] no_snapshot_gb_quota = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944946 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] num_iser_scan_tries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.944946 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] num_shell_tries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.945853 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] num_volume_device_scan_tries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.946358 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] nvmet_ns_id = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.946869 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] nvmet_port_id = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.947308 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_max_limit = 1000 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.947742 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_volume_ext_list = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.948182 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.948581 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_volume_listen = 0.0.0.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.948977 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_volume_listen_port = 8776 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.949340 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_volume_use_ssl = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.950828 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] osapi_volume_workers = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.950828 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] per_volume_size_limit = -1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.950828 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] periodic_fuzzy_delay = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.950828 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] periodic_interval = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.951393 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] project_id_regex = [0-9a-f\-]+ {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.951609 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] public_endpoint = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.952112 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] publish_errors = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.952493 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_backup_gigabytes = 1000 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.953004 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_backups = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.953372 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_consistencygroups = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.953771 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.954140 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_gigabytes = 1000 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.954496 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_groups = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.954905 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_snapshots = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.955291 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] quota_volumes = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.955673 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] rate_limit_burst = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.956032 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] rate_limit_except_level = CRITICAL {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.956504 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] rate_limit_interval = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.957027 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] replication_device = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.957804 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] report_discard_supported = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.958166 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] report_interval = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.959397 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] reservation_clean_interval = 86400 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.959833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] reservation_expire = 86400 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.960224 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] reserved_percentage = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.960658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.961066 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] restore_discard_excess_bytes = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.961451 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.961907 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] rpc_response_timeout = 120 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.962378 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] run_external_periodic_tasks = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.962925 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.963428 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] scst_target_driver = iscsi {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.963832 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] scst_target_iqn_name = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.964137 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_down_time = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.964536 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] snapshot_name_template = snapshot-%s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.964973 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] split_loggers = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.965276 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] state_path = /opt/stack/data/cinder {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.965663 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] storage_availability_zone = nova {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.966092 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] storage_protocol = iSCSI {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.966571 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] syslog_log_facility = LOG_USER {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.967089 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] target_helper = lioadm {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.967564 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] target_ip_address = 10.222.0.32 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.968564 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] target_port = 3260 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.969013 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.969419 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] target_protocol = iscsi {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.969775 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] trace_flags = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.970198 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] transfer_api_class = cinder.transfer.api.API {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.970588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] transport_url = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.971193 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] until_refresh = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.971550 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_chap_auth = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.971910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_default_quota_class = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.972414 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_eventlog = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.972793 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_journal = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.973352 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_json = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.973710 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_multipath_for_image_xfer = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.974287 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_stderr = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.974643 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] use_syslog = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.975051 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] verify_glance_signatures = enabled {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.975500 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_api_class = cinder.volume.api.API {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.975852 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_backend_name = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.976221 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_clear = zero {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.976671 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_clear_ionice = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.977181 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_clear_size = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.977591 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.977948 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_copy_bps_limit = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.978356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_dd_blocksize = 1M {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.978915 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.979378 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_name_template = volume-%s {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.979757 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volume_usage_audit_period = month {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.980192 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.980547 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] watch_log_file = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 07 21:36:35.981073 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.981539 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.981963 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_concurrency.disable_process_locking = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.982476 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.982945 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.983496 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.983944 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.984314 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.984679 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.985126 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.admin_password = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.985545 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.985923 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.admin_token = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.986280 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.admin_user = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.986853 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_admin_prefix = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.987384 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.987940 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_port = 35357 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.988345 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_protocol = https {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.988751 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_section = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.989155 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_type = password {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.989551 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_uri = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.989964 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.auth_version = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.990316 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.cache = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.990835 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.991264 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.991677 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.delay_auth_decision = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.992059 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.992523 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.http_connect_timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.992920 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.993277 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.identity_uri = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.993805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.include_service_catalog = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.994179 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.insecure = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.994551 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.interface = public {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.994954 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.995317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.995786 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.996170 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.996535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.997012 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.997480 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.997880 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.998230 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:35.998700 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.090788 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.region_name = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.090788 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.090788 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.service_token_roles_required = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.090788 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.service_type = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.090788 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.token_cache_time = 300 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.090788 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.backend = dogpile.cache.null {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.backend_argument = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.config_prefix = cache.oslo {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.dead_timeout = 60.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.debug_cache_backend = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.enable_retry_client = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091317 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.enable_socket_keepalive = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.enabled = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.expiration_time = 600 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.hashclient_retry_attempts = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.hashclient_retry_delay = 1.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_dead_retry = 300 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.091768 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_pool_maxsize = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.memcache_socket_timeout = 1.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.proxies = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.retry_attempts = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092201 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.retry_delay = 0.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.socket_keepalive_count = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.socket_keepalive_idle = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.socket_keepalive_interval = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.tls_allowed_ciphers = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.tls_cafile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.tls_certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.092658 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.tls_enabled = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093128 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.lockutils [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] Lock "cleanup_incomplete_backups_129416" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=129863) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} May 07 21:36:36.093213 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cache.tls_keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093292 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] Cleaning up incomplete backup operations. May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] coordination.backend_url = etcd3+http://10.222.0.32:2379 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.backend = sqlalchemy {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.connection = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.connection_debug = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.connection_parameters = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.connection_recycle_time = 3600 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093363 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.connection_trace = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.db_inc_retry_interval = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.db_max_retries = 20 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.db_max_retry_interval = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.db_retry_interval = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.max_overflow = 50 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.max_pool_size = 5 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.093805 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.max_retries = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.mysql_enable_ndb = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.pool_timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.retry_interval = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.slave_connection = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.sqlite_synchronous = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.094260 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] database.use_db_reconnect = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.enforce_new_defaults = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.enforce_scope = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.policy_default_rule = default {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.policy_file = policy.yaml {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097127 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.connection_string = messaging:// {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097535 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.enabled = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.es_doc_type = notification {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.es_scroll_size = 10000 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.es_scroll_time = 2m {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.filter_error_trace = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.hmac_keys = SECRET_KEY {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.sentinel_service_name = mymaster {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.097910 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.socket_timeout = 0.1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] profiler.trace_sqlalchemy = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] privsep_osbrick.capabilities = [21] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] privsep_osbrick.group = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] privsep_osbrick.helper_command = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098362 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] privsep_osbrick.user = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_sys_admin.group = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_sys_admin.helper_command = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] cinder_sys_admin.user = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.098833 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.auth_section = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.auth_type = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.cafile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.collect_timing = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.insecure = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099318 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.send_service_user_token = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099784 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.split_loggers = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099784 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] service_user.timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099784 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.backend_availability_zone = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099784 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.chap_password = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099784 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.chap_username = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.099784 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100184 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100184 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100184 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.driver_client_cert = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100184 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.driver_client_cert_key = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100184 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.driver_data_namespace = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100184 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100540 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100540 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.driver_use_ssl = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100540 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.enable_deferred_deletion = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100540 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.enable_unsupported_driver = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100540 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100540 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.filter_function = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100917 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.goodness_function = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100917 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100917 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100917 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100917 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.100917 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101309 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101309 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iscsi_iotype = fileio {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101309 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101309 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iscsi_target_flags = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101309 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iscsi_write_cache = on {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101309 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iser_helper = tgtadm {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101740 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iser_ip_address = 10.222.0.32 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101740 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iser_port = 3260 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101740 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101740 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101740 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.101740 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.num_shell_tries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.nvmet_port_id = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rados_connection_interval = 5 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rados_connection_retries = 3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102115 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_ceph_conf = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102574 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102574 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102574 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102574 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102574 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_pool = rbd {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.102574 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_secret_uuid = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107124 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107124 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.rbd_user = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107124 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107124 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.replication_device = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107124 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.report_discard_supported = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107124 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.reserved_percentage = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.scst_target_driver = iscsi {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.scst_target_iqn_name = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.storage_protocol = iSCSI {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.target_helper = tgtadm {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.107588 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.target_ip_address = 10.222.0.32 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108043 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.target_port = 3260 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108043 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108043 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.target_protocol = iscsi {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108043 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.trace_flags = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108043 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.unique_fqdn_network = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108043 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.use_chap_auth = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108487 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108487 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_backend_name = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108487 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_clear = zero {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108487 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_clear_ionice = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108487 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_clear_size = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.108487 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109016 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109016 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109016 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109016 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_reports.file_event_handler = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109016 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109016 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_reports.log_dir = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109456 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109456 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109456 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109456 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109456 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109456 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109906 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109906 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109906 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109906 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109906 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.109906 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110356 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110827 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110827 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110827 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110827 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110827 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.ssl = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.110827 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111267 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111267 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111267 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_rabbit.ssl_version = {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111267 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_notifications.driver = [] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111267 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_notifications.retry = -1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111267 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111704 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] oslo_messaging_notifications.transport_url = **** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111704 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111704 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111704 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.barbican_api_version = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111704 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.barbican_endpoint = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.111704 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.barbican_endpoint_type = public {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.barbican_region_name = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.cafile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.collect_timing = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.insecure = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112162 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.number_of_retries = 60 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.retry_delay = 1 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.send_service_user_token = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.split_loggers = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.verify_ssl = True {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican.verify_ssl_path = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.112699 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.auth_section = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.auth_type = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.cafile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.collect_timing = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.insecure = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113211 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.split_loggers = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] barbican_service_user.timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.approle_role_id = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.approle_secret_id = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.cafile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.certfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.collect_timing = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.113781 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.insecure = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.keyfile = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.kv_mountpoint = secret {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.kv_version = 2 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.namespace = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.root_token_id = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.split_loggers = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114216 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.ssl_ca_crt_file = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114873 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.timeout = None {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114873 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.use_ssl = False {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114873 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 07 21:36:36.114873 ncdv-master-1 cinder-backup[129416]: DEBUG oslo_service.service [None req-ab5e6729-44b5-4287-b50c-9dfa6b69f653 None None] ******************************************************************************** {{(pid=129416) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} May 07 21:36:36.147098 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 21:36:36.190955 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.lockutils [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] Lock "cleanup_incomplete_backups_129416" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.184s {{(pid=129863) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} May 07 21:36:36.198285 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 21:36:36.205770 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:36:36.239618 ncdv-master-1 cinder-backup[129863]: INFO cinder.keymgr.migration [None req-b0621b14-b8cb-4526-a979-0ee988d88457 None None] Not migrating encryption keys because the ConfKeyManager is still in use. May 07 21:36:36.260389 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.service [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] Creating RPC server for service cinder-backup {{(pid=129863) start /opt/stack/cinder/cinder/service.py:231}} May 07 21:36:36.270976 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.service [None req-d97255d8-25d9-4a16-82d3-6d75403509ec None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=129863) start /opt/stack/cinder/cinder/service.py:237}} May 07 21:37:47.299638 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:37:47.300375 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:38:47.295538 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:38:47.296224 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:39:47.294239 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:39:47.294918 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:40:47.290724 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:40:47.291395 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:41:47.287668 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:41:47.288300 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:42:47.292341 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:42:47.292991 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:43:47.297065 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:43:47.297626 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:44:47.302127 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:44:47.303042 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:45:47.298200 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:45:47.298807 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:46:47.295923 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:46:47.297172 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:47:47.293987 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:47:47.295569 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:48:47.301102 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:48:47.305525 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:50:47.310596 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:50:47.311738 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:51:47.315616 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:51:47.316784 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:52:47.315476 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:52:47.316465 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:53:47.313306 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:53:47.315388 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:54:47.321250 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:54:47.321858 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:55:47.320132 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:55:47.320820 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:56:47.316461 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:56:47.317145 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:57:47.314529 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:57:47.315475 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:58:47.320144 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:58:47.321354 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 21:59:47.317165 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 21:59:47.317991 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:00:47.318281 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:00:47.320538 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:01:47.320288 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:01:47.321149 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:02:47.317505 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:02:47.318219 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:03:03.043206 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Create backup started, backup: 44335655-37f1-4718-9305-8b11c79b2434 volume: 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8. May 07 22:03:03.052453 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:03:02Z,data_timestamp=2022-05-07T22:03:02Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1434947356',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=44335655-37f1-4718-9305-8b11c79b2434,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='9efc1fbca3ab413fabfc108b752dd251',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='e1d72e319762436a9a2f8a69e5402bf6',volume_id=4c2b93f0-2cae-4c4a-8f15-32043b50e7d8) May 07 22:03:03.119570 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:03.126891 ncdv-master-1 cinder-backup[129863]: INFO oslo.privsep.daemon [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpo66f1bmf/privsep.sock'] May 07 22:03:03.298544 ncdv-master-1 sudo[155486]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpo66f1bmf/privsep.sock May 07 22:03:03.319219 ncdv-master-1 sudo[155486]: pam_unix(sudo:session): session opened for user root by (uid=0) May 07 22:03:06.514084 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-45e49f68-24c9-47da-97eb-543f2ff8c192 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:08.631190 ncdv-master-1 sudo[155486]: pam_unix(sudo:session): session closed for user root May 07 22:03:08.650378 ncdv-master-1 cinder-backup[129863]: INFO oslo.privsep.daemon [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Spawned new privsep daemon via rootwrap May 07 22:03:08.650953 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Accepted privsep connection to /tmp/tmpo66f1bmf/privsep.sock {{(pid=129863) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} May 07 22:03:08.652602 ncdv-master-1 cinder-backup[129863]: INFO oslo.privsep.daemon [-] privsep daemon starting May 07 22:03:08.653299 ncdv-master-1 cinder-backup[129863]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 May 07 22:03:08.653573 ncdv-master-1 cinder-backup[129863]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none May 07 22:03:08.654061 ncdv-master-1 cinder-backup[129863]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 155525 May 07 22:03:08.656013 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[f3984725-6172-48bf-8d7a-cfa8209777aa]: (2,) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.106746 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.136107 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.029s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:09.137127 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[e46ae334-50f1-4ada-95d6-c403b2cd28ec]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.143314 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.153501 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[449fe422-550d-4d03-8f28-95a2492cd06d]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:03:09.175032 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:03:09.176323 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[449fe422-550d-4d03-8f28-95a2492cd06d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.178523 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.186758 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6cc6b932-6926-46b3-8e83-b1971c68cb28]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:03:09.191026 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:03:09.192702 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[6cc6b932-6926-46b3-8e83-b1971c68cb28]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.204865 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.274547 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.079s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:09.275113 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[d657c1ef-61d5-48eb-9414-d878cd27455a]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.276744 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.360618 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.083s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:09.361139 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[8de86cac-3c0e-4a5c-8415-899ee8af005e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.365730 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.394694 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.031s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:09.395009 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[4d098299-8a51-4016-ac7c-7a0c933fbe37]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:09.397410 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:09.479207 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] CMD "nvme version" returned: 0 in 0.083s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:09.510517 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:03:09.510702 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:03:09.511332 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:03:09.512587 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] <== get_connector_properties: return (6391ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:10.969678 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:03:10.973103 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4c2b93f0-2cae-4c4a-8f15-32043b50e7d8', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:10.976059 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:03:11.103707 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] <== connect_volume: return (129ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:11.143661 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Starting backup of volume='4c2b93f0-2cae-4c4a-8f15-32043b50e7d8'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:03:11.144625 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Volume file is RBD: attempting optimized backup {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 07 22:03:11.220296 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Creating base image 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' {{(pid=129863) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 07 22:03:11.221135 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:03:13.038966 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Using --from-snap 'None' for incremental backup of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8. {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 07 22:03:13.043292 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Creating backup snapshot='backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266' {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 07 22:03:14.041104 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Performing differential transfer from 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8' to 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:03:14.041795 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_gu0ou8zk --pool volumes volumes/volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8@backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:03:14.041899 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:03:15.077790 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Differential backup transfer completed in 1.0357s {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 07 22:03:15.078488 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:03:15.097497 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:03:15.100578 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:03:15.125077 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:03:15.127678 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:03:15.150022 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] No metadata type 'volume-glance-metadata' available {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 07 22:03:15.150762 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Backing up metadata for volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:03:15.264583 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Backup '44335655-37f1-4718-9305-8b11c79b2434' of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8 finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:03:15.266331 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4c2b93f0-2cae-4c4a-8f15-32043b50e7d8', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:15.319950 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] <== disconnect_volume: return (52ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:15.513903 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-b081dca8-a479-4d1e-b998-17265761e68a tempest-VolumesBackupsAdminTest-1667063411 None] Create backup finished. backup: 44335655-37f1-4718-9305-8b11c79b2434. May 07 22:03:16.926183 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-1e7c338f-ee29-4a00-b2ff-7ad25aac4892 tempest-VolumesBackupsAdminTest-667405632 None] Export record started, backup: 44335655-37f1-4718-9305-8b11c79b2434. May 07 22:03:16.929377 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-1e7c338f-ee29-4a00-b2ff-7ad25aac4892 tempest-VolumesBackupsAdminTest-667405632 None] Export record finished, backup 44335655-37f1-4718-9305-8b11c79b2434 exported. May 07 22:03:17.122070 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-1b653d65-225c-4190-b486-ea622d560b75 tempest-VolumesBackupsAdminTest-667405632 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiZGVmZGM3NjktMzEzNS00YjFlLWJmMzctOTM4YzJlZDYwNGI2IiwgInVzZXJfaWQiOiAiZTFkNzJlMzE5NzYyNDM2YTlhMmY4YTY5ZTU0MDJiZjYiLCAicHJvamVjdF9pZCI6ICI5ZWZjMWZiY2EzYWI0MTNmYWJmYzEwOGI3NTJkZDI1MSIsICJ2b2x1bWVfaWQiOiAiNGMyYjkzZjAtMmNhZS00YzRhLThmMTUtMzIwNDNiNTBlN2Q4IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xNDM0OTQ3MzU2IiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLTRjMmI5M2YwLTJjYWUtNGM0YS04ZjE1LTMyMDQzYjUwZTdkOC5iYWNrdXAuNDQzMzU2NTUtMzdmMS00NzE4LTkzMDUtOGIxMWM3OWIyNDM0XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wNS0wN1QyMjowMzowM1oiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wNS0wN1QyMjowMzowM1oiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTA1LTA3VDIyOjAzOjE2WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. May 07 22:03:17.145337 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-1b653d65-225c-4190-b486-ea622d560b75 tempest-VolumesBackupsAdminTest-667405632 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:17.282020 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-1b653d65-225c-4190-b486-ea622d560b75 tempest-VolumesBackupsAdminTest-667405632 None] Import record id defdc769-3135-4b1e-bf37-938c2ed604b6 metadata from driver finished. May 07 22:03:19.862621 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Restore backup started, backup: 44335655-37f1-4718-9305-8b11c79b2434 volume: 80906e2f-de30-4b36-a909-d686d31082b3. May 07 22:03:19.898474 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:19.907476 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:19.918078 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.018s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:19.918389 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[ccfc7a1f-87ea-4de1-995c-168ed8dbd00b]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:19.922086 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3348c6d0-fe57-4252-bc34-5da53a4a64b6]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:03:19.933690 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[3348c6d0-fe57-4252-bc34-5da53a4a64b6]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:19.935897 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:19.976045 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[af8ca8eb-552c-43d8-9993-0e42820bd82f]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:03:19.977004 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:03:19.978268 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[af8ca8eb-552c-43d8-9993-0e42820bd82f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:19.979804 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:20.021690 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.042s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:20.021982 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[44f37fc9-1e42-4217-9f08-781b3b141038]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:20.024030 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:20.048484 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.024s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:20.048808 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[66d2fe6f-8a86-4b30-bd95-14b09d994495]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:20.051220 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:20.083136 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.032s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:20.083471 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[fcc7c164-89a5-4047-8ac7-82536ae35229]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:20.084293 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:20.141032 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] CMD "nvme version" returned: 0 in 0.056s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:20.143821 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:03:20.144206 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:03:20.144584 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:03:20.145282 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] <== get_connector_properties: return (246ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:21.219089 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:03:21.222554 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-80906e2f-de30-4b36-a909-d686d31082b3', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '80906e2f-de30-4b36-a909-d686d31082b3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:21.224906 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:03:21.380077 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] <== connect_volume: return (156ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:21.388832 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:21.403731 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Starting restore from Ceph backup=44335655-37f1-4718-9305-8b11c79b2434 to volume=volume-80906e2f-de30-4b36-a909-d686d31082b3 {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 07 22:03:21.575428 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Looking for snapshot of backup base 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:03:21.576358 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Found snapshot 'backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:03:21.585943 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Volume file is RBD. {{(pid=129863) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 07 22:03:21.656241 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Attempting differential restore. {{(pid=129863) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 07 22:03:21.657323 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Attempting incremental restore from base='volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' snap='backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266' {{(pid=129863) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 07 22:03:21.657931 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Performing differential transfer from 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' to 'volume-80906e2f-de30-4b36-a909-d686d31082b3' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:03:21.658570 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434@backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:03:21.659221 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_bm5_iloi --pool volumes - volumes/volume-80906e2f-de30-4b36-a909-d686d31082b3' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:03:23.245949 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Restore transfer completed in 1.5875s {{(pid=129863) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 07 22:03:23.247204 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 07 22:03:23.340509 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Restoring volume base metadata {{(pid=129863) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 07 22:03:23.397563 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] No metadata of type 'volume-metadata' to restore {{(pid=129863) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 07 22:03:23.398246 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=129863) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 07 22:03:23.405577 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Restore to volume 80906e2f-de30-4b36-a909-d686d31082b3 finished successfully. {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 07 22:03:23.407731 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-80906e2f-de30-4b36-a909-d686d31082b3', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '80906e2f-de30-4b36-a909-d686d31082b3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:23.436263 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] <== disconnect_volume: return (27ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:23.513122 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.manager [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Encryption key ID for volume 80906e2f-de30-4b36-a909-d686d31082b3 already matches encryption key ID in backup 44335655-37f1-4718-9305-8b11c79b2434. {{(pid=129863) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 07 22:03:23.640502 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-98f01482-5dd0-47cd-b4f0-e21097a158d4 tempest-VolumesBackupsAdminTest-1667063411 None] Finished restoring backup 44335655-37f1-4718-9305-8b11c79b2434 to volume 80906e2f-de30-4b36-a909-d686d31082b3. May 07 22:03:24.576273 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Delete backup started, backup: defdc769-3135-4b1e-bf37-938c2ed604b6. May 07 22:03:24.580205 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Delete started for backup=defdc769-3135-4b1e-bf37-938c2ed604b6 {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:03:24.580821 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Trying diff format basename='volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' for backup base image of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:03:24.688274 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Looking for snapshot of backup base 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:03:24.689059 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Backup 'defdc769-3135-4b1e-bf37-938c2ed604b6' has no snapshot {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:962}} May 07 22:03:24.689803 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] No backup snapshot to delete {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} May 07 22:03:24.715930 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Backup base image of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8 still has 1 snapshots so skipping base image delete. May 07 22:03:24.806044 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Metadata backup object 'backup.defdc769-3135-4b1e-bf37-938c2ed604b6.meta' not found - ignoring {{(pid=129863) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:162}} May 07 22:03:24.810278 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Delete of backup 'defdc769-3135-4b1e-bf37-938c2ed604b6' for volume '4c2b93f0-2cae-4c4a-8f15-32043b50e7d8' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:03:24.820817 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:24.975275 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Created reservations ['9cd31620-a9f3-42f2-b950-3c2bc94e43e3', '30fdfbfc-3804-44e7-a219-30e1c717e5e4'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:03:25.077203 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-0a0d4034-f42a-473f-9c62-772fe83d0a5c tempest-VolumesBackupsAdminTest-667405632 None] Delete backup finished, backup defdc769-3135-4b1e-bf37-938c2ed604b6 deleted. May 07 22:03:25.759821 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Delete backup started, backup: 44335655-37f1-4718-9305-8b11c79b2434. May 07 22:03:25.759821 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Delete started for backup=44335655-37f1-4718-9305-8b11c79b2434 {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:03:25.760457 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Trying diff format basename='volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' for backup base image of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:03:25.875004 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Looking for snapshot of backup base 'volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:03:25.875756 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Found snapshot 'backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:03:25.876441 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Deleting backup snapshot='backup.44335655-37f1-4718-9305-8b11c79b2434.snap.1651960993.0403266' {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 07 22:03:26.391822 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Deleting backup base image='volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8. May 07 22:03:26.513897 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-45e49f68-24c9-47da-97eb-543f2ff8c192 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:26.875381 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Base backup image='volume-4c2b93f0-2cae-4c4a-8f15-32043b50e7d8.backup.44335655-37f1-4718-9305-8b11c79b2434' of volume 4c2b93f0-2cae-4c4a-8f15-32043b50e7d8 deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:03:26.964463 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Delete of backup '44335655-37f1-4718-9305-8b11c79b2434' for volume '4c2b93f0-2cae-4c4a-8f15-32043b50e7d8' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:03:27.034536 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Created reservations ['fabc17a8-d04f-4043-8ee3-d7da696c4e00', '604e5c9f-d13d-43dd-bbcb-34df5b984d9f'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:03:27.093511 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-f1a5e4cf-9602-47d4-8e21-f0dae483e821 tempest-VolumesBackupsAdminTest-1667063411 None] Delete backup finished, backup 44335655-37f1-4718-9305-8b11c79b2434 deleted. May 07 22:03:29.807932 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Create backup started, backup: 24077eb6-07be-49ca-9664-9b299764221a volume: 7fd9a46a-c86c-4195-9beb-9b650657275c. May 07 22:03:29.830797 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:03:29Z,data_timestamp=2022-05-07T22:03:29Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-2091994386',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=24077eb6-07be-49ca-9664-9b299764221a,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='9efc1fbca3ab413fabfc108b752dd251',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='e1d72e319762436a9a2f8a69e5402bf6',volume_id=7fd9a46a-c86c-4195-9beb-9b650657275c) May 07 22:03:29.932071 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:29.934013 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:29.966248 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.031s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:29.967104 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[6e6ae7c7-c337-42a3-9774-8a1e13252534]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:29.971399 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:30.011477 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3f01d9ae-9829-45ae-bccc-ae3fd0afd413]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:03:30.011900 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:03:30.013358 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[3f01d9ae-9829-45ae-bccc-ae3fd0afd413]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:30.016622 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:30.032548 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7631c1a5-6725-4e70-9847-bb32d6be488f]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:03:30.033144 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:03:30.034661 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[7631c1a5-6725-4e70-9847-bb32d6be488f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:30.035232 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:30.046176 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.011s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:30.047148 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[2cd2d568-6643-4b42-88eb-da4a40263511]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:30.049376 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:30.062432 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.013s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:30.063078 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[da907868-b7dc-4770-81f2-26f9a96c823f]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:30.067316 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:30.100735 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.035s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:30.101030 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[41ed3245-21fd-4b80-81f4-d6fa7348120a]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:03:30.101786 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:03:30.157917 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] CMD "nvme version" returned: 0 in 0.055s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:03:30.159845 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:03:30.160222 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:03:30.160979 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:03:30.162172 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] <== get_connector_properties: return (228ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:31.362395 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:03:31.364825 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-7fd9a46a-c86c-4195-9beb-9b650657275c', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7fd9a46a-c86c-4195-9beb-9b650657275c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:31.366391 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:03:31.458754 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] <== connect_volume: return (93ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:31.470580 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:31.499571 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Starting backup of volume='7fd9a46a-c86c-4195-9beb-9b650657275c'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:03:31.500537 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Volume file is RBD: attempting optimized backup {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 07 22:03:31.544259 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Creating base image 'volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' {{(pid=129863) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 07 22:03:31.545052 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:03:31.596114 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Using --from-snap 'None' for incremental backup of volume 7fd9a46a-c86c-4195-9beb-9b650657275c. {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 07 22:03:31.596663 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Creating backup snapshot='backup.24077eb6-07be-49ca-9664-9b299764221a.snap.1651961011.596145' {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 07 22:03:31.972465 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Performing differential transfer from 'volume-7fd9a46a-c86c-4195-9beb-9b650657275c' to 'volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:03:31.975015 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_qlgv8cyy --pool volumes volumes/volume-7fd9a46a-c86c-4195-9beb-9b650657275c@backup.24077eb6-07be-49ca-9664-9b299764221a.snap.1651961011.596145 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:03:31.976311 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:03:33.112805 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Differential backup transfer completed in 1.1399s {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 07 22:03:33.114139 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:03:33.141336 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:03:33.144361 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:03:33.165548 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:03:33.166157 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:03:33.183540 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] No metadata type 'volume-glance-metadata' available {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 07 22:03:33.184102 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Backing up metadata for volume 7fd9a46a-c86c-4195-9beb-9b650657275c. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:03:33.323113 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Backup '24077eb6-07be-49ca-9664-9b299764221a' of volume 7fd9a46a-c86c-4195-9beb-9b650657275c finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:03:33.325349 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-7fd9a46a-c86c-4195-9beb-9b650657275c', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7fd9a46a-c86c-4195-9beb-9b650657275c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:03:33.379986 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] <== disconnect_volume: return (53ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:03:33.519645 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-7ceb5d1f-1508-4f6e-85ce-61f2c3c7d358 tempest-VolumesBackupsAdminTest-1667063411 None] Create backup finished. backup: 24077eb6-07be-49ca-9664-9b299764221a. May 07 22:03:34.361593 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-858157b0-f3c9-4334-a50f-7ec765c7897a tempest-VolumesBackupsAdminTest-667405632 None] Reset backup status started, backup_id: 24077eb6-07be-49ca-9664-9b299764221a, status: error. May 07 22:03:34.362731 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-858157b0-f3c9-4334-a50f-7ec765c7897a tempest-VolumesBackupsAdminTest-667405632 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. May 07 22:03:35.522584 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Delete backup started, backup: 24077eb6-07be-49ca-9664-9b299764221a. May 07 22:03:35.527431 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Delete started for backup=24077eb6-07be-49ca-9664-9b299764221a {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:03:35.527883 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Trying diff format basename='volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' for backup base image of volume 7fd9a46a-c86c-4195-9beb-9b650657275c. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:03:35.614989 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Looking for snapshot of backup base 'volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:03:35.615548 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Found snapshot 'backup.24077eb6-07be-49ca-9664-9b299764221a.snap.1651961011.596145' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:03:35.616560 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Deleting backup snapshot='backup.24077eb6-07be-49ca-9664-9b299764221a.snap.1651961011.596145' {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 07 22:03:36.194816 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Deleting backup base image='volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' of volume 7fd9a46a-c86c-4195-9beb-9b650657275c. May 07 22:03:36.517463 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-45e49f68-24c9-47da-97eb-543f2ff8c192 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:03:36.773606 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Base backup image='volume-7fd9a46a-c86c-4195-9beb-9b650657275c.backup.24077eb6-07be-49ca-9664-9b299764221a' of volume 7fd9a46a-c86c-4195-9beb-9b650657275c deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:03:36.851633 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Delete of backup '24077eb6-07be-49ca-9664-9b299764221a' for volume '7fd9a46a-c86c-4195-9beb-9b650657275c' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:03:36.913292 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Created reservations ['2ace7352-2a0d-414d-aa3a-fd651e0d642e', '490c03fb-74eb-4f97-82b6-f1e62018da8b'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:03:36.987607 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-a719f158-c6ab-4175-9845-34f037f476da tempest-VolumesBackupsAdminTest-1667063411 None] Delete backup finished, backup 24077eb6-07be-49ca-9664-9b299764221a deleted. May 07 22:03:47.320723 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:03:47.321344 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:04:47.317731 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:04:47.318296 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:05:47.314833 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:05:47.315527 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:06:47.312681 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:06:47.313274 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:07:47.311861 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:07:47.313324 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:08:17.761932 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Create backup started, backup: c003f529-0a51-47c4-b662-aeae7a5cf4dc volume: 28875749-3b9f-4237-b844-689c84a5f03f snapshot: f269fe76-0736-470f-aa1e-6e252c4dee4e. May 07 22:08:17.782663 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:08:17Z,data_timestamp=2022-05-07T22:08:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-325745070',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=c003f529-0a51-47c4-b662-aeae7a5cf4dc,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='72be56c2d4a54014a8f741c451fe014e',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='f269fe76-0736-470f-aa1e-6e252c4dee4e',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='97ef9a3039534d90ba9d268970cbe9d4',volume_id=28875749-3b9f-4237-b844-689c84a5f03f) May 07 22:08:18.556736 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:08:18.563238 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.599898 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.041s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:08:18.600221 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[5c30e16e-52e9-4ab0-bf56-9151114d4c97]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:08:18.601583 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.620341 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a5d6eec4-c268-409c-b2ff-dc6d2bb462c8]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:08:18.622168 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:08:18.624384 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[a5d6eec4-c268-409c-b2ff-dc6d2bb462c8]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:08:18.629302 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.637803 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ceec42c3-41d7-49fc-aa14-ab4c7d119037]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:08:18.638551 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:08:18.641375 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[ceec42c3-41d7-49fc-aa14-ab4c7d119037]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:08:18.644322 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.710749 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.067s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:08:18.712233 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[683af49a-6d00-4d89-8935-55a5856906c9]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:08:18.715396 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.748186 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.033s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:08:18.748621 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[23cba6c8-7418-4179-8fe9-84e02f7089a5]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:08:18.751172 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.764628 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.014s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:08:18.765833 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[73d5bcc3-b16f-4645-bd55-25c637e9a089]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:08:18.767429 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:08:18.808980 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] CMD "nvme version" returned: 0 in 0.040s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:08:18.813280 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:08:18.814729 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:08:18.815407 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:08:18.816293 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] <== get_connector_properties: return (259ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:08:20.187808 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:08:20.194027 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fb5e7fa1-3cf2-4391-b44a-3169654739e7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fb5e7fa1-3cf2-4391-b44a-3169654739e7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:08:20.196780 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:08:20.349233 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] <== connect_volume: return (154ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:08:20.361895 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:08:20.405586 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Starting backup of volume='28875749-3b9f-4237-b844-689c84a5f03f'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:08:20.471486 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:08:20.472165 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Creating backup base image='volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.base' for volume 28875749-3b9f-4237-b844-689c84a5f03f. {{(pid=129863) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:875}} May 07 22:08:20.568050 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Copying data from volume 28875749-3b9f-4237-b844-689c84a5f03f. {{(pid=129863) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:887}} May 07 22:08:20.598590 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferring data between 'volume-28875749-3b9f-4237-b844-689c84a5f03f' and 'volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.base' {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:417}} May 07 22:08:20.599578 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] 8 chunks of 134217728 bytes to be transferred {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:421}} May 07 22:08:23.001781 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 1 of 8 (54605K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:25.539902 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 2 of 8 (51667K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:27.546470 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 3 of 8 (65375K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:29.424752 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 4 of 8 (69839K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:31.226844 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 5 of 8 (72758K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:33.275802 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 6 of 8 (63988K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:35.609634 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 7 of 8 (56178K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:37.723866 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Transferred chunk 8 of 8 (62011K/s) {{(pid=129863) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 07 22:08:37.814928 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:08:37.836854 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:08:37.837273 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:08:37.851658 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:08:37.852037 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:08:37.868224 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] No metadata type 'volume-glance-metadata' available {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 07 22:08:37.868808 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Backing up metadata for volume 28875749-3b9f-4237-b844-689c84a5f03f. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:08:37.966037 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Backup 'c003f529-0a51-47c4-b662-aeae7a5cf4dc' of volume 28875749-3b9f-4237-b844-689c84a5f03f finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:08:37.967575 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fb5e7fa1-3cf2-4391-b44a-3169654739e7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fb5e7fa1-3cf2-4391-b44a-3169654739e7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:08:37.998897 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] <== disconnect_volume: return (29ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:08:38.231674 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-c167969e-cf8c-4534-9947-92b94b05ed91 tempest-VolumesSnapshotTestJSON-1726756463 None] Create backup finished. backup: c003f529-0a51-47c4-b662-aeae7a5cf4dc. May 07 22:08:39.162575 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Delete backup started, backup: c003f529-0a51-47c4-b662-aeae7a5cf4dc. May 07 22:08:39.165381 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Delete started for backup=c003f529-0a51-47c4-b662-aeae7a5cf4dc {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:08:39.166259 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Trying diff format basename='volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.c003f529-0a51-47c4-b662-aeae7a5cf4dc' for backup base image of volume 28875749-3b9f-4237-b844-689c84a5f03f. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:08:39.243572 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Image 'volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.c003f529-0a51-47c4-b662-aeae7a5cf4dc' not found - trying diff format name {{(pid=129863) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:693}} May 07 22:08:39.289827 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Looking for snapshot of backup base 'volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.base' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:08:39.290537 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Backup base 'volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.base' has no snapshots {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:955}} May 07 22:08:39.292725 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] No backup snapshot to delete {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} May 07 22:08:39.313607 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Deleting backup base image='volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.base' of volume 28875749-3b9f-4237-b844-689c84a5f03f. May 07 22:08:40.592715 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Base backup image='volume-28875749-3b9f-4237-b844-689c84a5f03f.backup.base' of volume 28875749-3b9f-4237-b844-689c84a5f03f deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:08:40.696348 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Delete of backup 'c003f529-0a51-47c4-b662-aeae7a5cf4dc' for volume '28875749-3b9f-4237-b844-689c84a5f03f' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:08:40.706487 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:08:40.751820 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Created reservations ['ba7d3964-6b8d-4a6c-a6a3-fa0c3284e858', '626b02c9-545c-42f8-8508-fa9d16733ae1'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:08:40.826405 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-4f179325-02ed-4cfb-a790-ef42d42a4e27 tempest-VolumesSnapshotTestJSON-1726756463 None] Delete backup finished, backup c003f529-0a51-47c4-b662-aeae7a5cf4dc deleted. May 07 22:08:47.312215 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:08:47.313605 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:09:00.063457 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Create backup started, backup: fa1cebef-76a3-4a6a-9a02-031f6821fa00 volume: aeca4df0-304c-4201-9f83-f45ab5b82bd9. May 07 22:09:00.096736 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:08:59Z,data_timestamp=2022-05-07T22:08:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1809999679',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=fa1cebef-76a3-4a6a-9a02-031f6821fa00,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='e2bec96bf83d4fb296cbb8aeb28d576e',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='38e6d754538146aea01b0f51c8e2cc2f',volume_id=aeca4df0-304c-4201-9f83-f45ab5b82bd9) May 07 22:09:00.190496 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:00.193889 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.218212 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.026s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:00.218726 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[53b36d8a-eb18-4885-ac01-f0605616ef63]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:00.222684 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.253316 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1077e0cf-0e16-4b4b-b422-9eb9d111847f]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:00.254018 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:00.256067 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[1077e0cf-0e16-4b4b-b422-9eb9d111847f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:00.257726 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.281966 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[12dfcae4-6355-4cd9-bcc4-1e54de767e02]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:00.282263 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:00.285046 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[12dfcae4-6355-4cd9-bcc4-1e54de767e02]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:00.287228 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.322465 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.035s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:00.323656 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[fea6f1b8-f3b0-40f6-994f-4e7f98ce82b2]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:00.328544 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.359487 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.033s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:00.359780 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[609c2b62-235a-46ef-9dec-5c4567f1a829]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:00.366210 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.390783 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.027s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:00.391731 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[6a8f1193-5d2a-4327-a674-86215cc39469]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:00.392493 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:00.462637 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] CMD "nvme version" returned: 0 in 0.070s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:00.465770 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:09:00.466298 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:09:00.466960 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:09:00.478528 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] <== get_connector_properties: return (276ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:02.524187 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:09:02.526362 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'aeca4df0-304c-4201-9f83-f45ab5b82bd9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:02.527451 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:09:02.648635 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] <== connect_volume: return (121ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:02.657572 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:09:02.714125 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Starting backup of volume='aeca4df0-304c-4201-9f83-f45ab5b82bd9'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:09:02.715394 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Volume file is RBD: attempting optimized backup {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 07 22:09:02.799356 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Creating base image 'volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' {{(pid=129863) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 07 22:09:02.800219 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:09:02.907684 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Using --from-snap 'None' for incremental backup of volume aeca4df0-304c-4201-9f83-f45ab5b82bd9. {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 07 22:09:02.908220 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Creating backup snapshot='backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00.snap.1651961342.907715' {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 07 22:09:03.812427 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Performing differential transfer from 'volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9' to 'volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:09:03.813187 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_f1o3aopv --pool volumes volumes/volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9@backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00.snap.1651961342.907715 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:09:03.813701 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:09:04.887116 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Differential backup transfer completed in 1.0742s {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 07 22:09:04.888326 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:09:04.936671 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:09:04.939020 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:09:04.958004 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:09:04.958752 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:09:04.974313 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] No metadata type 'volume-glance-metadata' available {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 07 22:09:04.976268 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Backing up metadata for volume aeca4df0-304c-4201-9f83-f45ab5b82bd9. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:09:05.054598 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Backup 'fa1cebef-76a3-4a6a-9a02-031f6821fa00' of volume aeca4df0-304c-4201-9f83-f45ab5b82bd9 finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:09:05.057618 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'aeca4df0-304c-4201-9f83-f45ab5b82bd9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:05.079429 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] <== disconnect_volume: return (19ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:05.228324 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-d5f04209-3b70-48a1-b0fc-f615c459bce5 tempest-VolumesBackupsTest-1817505426 None] Create backup finished. backup: fa1cebef-76a3-4a6a-9a02-031f6821fa00. May 07 22:09:05.498360 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Delete backup started, backup: fa1cebef-76a3-4a6a-9a02-031f6821fa00. May 07 22:09:05.501771 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Delete started for backup=fa1cebef-76a3-4a6a-9a02-031f6821fa00 {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:09:05.502255 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Trying diff format basename='volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' for backup base image of volume aeca4df0-304c-4201-9f83-f45ab5b82bd9. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:09:05.615785 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Looking for snapshot of backup base 'volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:09:05.617079 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Found snapshot 'backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00.snap.1651961342.907715' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:09:05.617746 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Deleting backup snapshot='backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00.snap.1651961342.907715' {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 07 22:09:06.533930 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-45e49f68-24c9-47da-97eb-543f2ff8c192 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:09:06.601509 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Deleting backup base image='volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' of volume aeca4df0-304c-4201-9f83-f45ab5b82bd9. May 07 22:09:07.124694 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Base backup image='volume-aeca4df0-304c-4201-9f83-f45ab5b82bd9.backup.fa1cebef-76a3-4a6a-9a02-031f6821fa00' of volume aeca4df0-304c-4201-9f83-f45ab5b82bd9 deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:09:07.206506 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Delete of backup 'fa1cebef-76a3-4a6a-9a02-031f6821fa00' for volume 'aeca4df0-304c-4201-9f83-f45ab5b82bd9' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:09:07.259023 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Created reservations ['16779be6-9765-4ab6-8755-e8f0dab48f96', 'd5fc2664-6334-4953-817d-f65aca462446'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:09:07.309298 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-ff7fa230-514c-4eb1-ad5b-13df9b57ffb3 tempest-VolumesBackupsTest-1817505426 None] Delete backup finished, backup fa1cebef-76a3-4a6a-9a02-031f6821fa00 deleted. May 07 22:09:20.357117 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Create backup started, backup: e44184b1-0f9f-4e9c-a865-dd57d2a803d0 volume: ee967edb-ea50-423b-8d88-03897f067e75. May 07 22:09:20.396947 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:09:20Z,data_timestamp=2022-05-07T22:09:20Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-601449591',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e44184b1-0f9f-4e9c-a865-dd57d2a803d0,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='e2bec96bf83d4fb296cbb8aeb28d576e',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='38e6d754538146aea01b0f51c8e2cc2f',volume_id=ee967edb-ea50-423b-8d88-03897f067e75) May 07 22:09:20.511034 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:20.513053 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.564842 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.051s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:20.565474 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[dd1b58c1-d8f4-4030-ba99-21bd3841f7c2]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:20.571204 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.591087 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[176b1be0-7d9e-47b3-8ed9-8e59761efa13]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:20.591434 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:20.593451 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[176b1be0-7d9e-47b3-8ed9-8e59761efa13]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:20.602878 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.637148 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f0113289-84f4-4007-bf6b-5a3f8ef4d63b]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:20.639080 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:20.642486 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[f0113289-84f4-4007-bf6b-5a3f8ef4d63b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:20.645919 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.680510 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.036s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:20.681255 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[5bae6d88-73d4-4598-a204-261ae3e50aec]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:20.686391 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.718213 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.034s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:20.718668 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[16ab86bf-c5c8-4506-b45b-3ea5c7c79dfd]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:20.720994 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.749121 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.028s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:20.749714 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[14613f08-141f-4def-8321-52c234f3a7a9]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:20.750655 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:20.800686 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] CMD "nvme version" returned: 0 in 0.050s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:20.802746 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:09:20.803444 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:09:20.804006 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:09:20.805080 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] <== get_connector_properties: return (292ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:21.998449 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:09:22.001346 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ee967edb-ea50-423b-8d88-03897f067e75', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ee967edb-ea50-423b-8d88-03897f067e75', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:22.003907 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:09:22.155658 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] <== connect_volume: return (153ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:22.166008 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:09:22.206593 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Starting backup of volume='ee967edb-ea50-423b-8d88-03897f067e75'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:09:22.209153 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Volume file is RBD: attempting optimized backup {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 07 22:09:22.313023 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Creating base image 'volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' {{(pid=129863) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 07 22:09:22.313984 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:09:22.396524 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Using --from-snap 'None' for incremental backup of volume ee967edb-ea50-423b-8d88-03897f067e75. {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 07 22:09:22.396942 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Creating backup snapshot='backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526' {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 07 22:09:22.574757 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Performing differential transfer from 'volume-ee967edb-ea50-423b-8d88-03897f067e75' to 'volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:09:22.575201 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_ucf387bo --pool volumes volumes/volume-ee967edb-ea50-423b-8d88-03897f067e75@backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:09:22.575585 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:09:31.610838 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Differential backup transfer completed in 9.0358s {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 07 22:09:31.611999 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:09:31.629984 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:09:31.630641 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:09:31.638442 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:09:31.639079 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:09:31.647670 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 07 22:09:31.648496 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Backing up metadata for volume ee967edb-ea50-423b-8d88-03897f067e75. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:09:31.741112 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Backup 'e44184b1-0f9f-4e9c-a865-dd57d2a803d0' of volume ee967edb-ea50-423b-8d88-03897f067e75 finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:09:31.746106 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ee967edb-ea50-423b-8d88-03897f067e75', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ee967edb-ea50-423b-8d88-03897f067e75', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:31.779061 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] <== disconnect_volume: return (31ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:31.871563 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-e44fb534-9fac-4cd5-9872-f97c96c89f7b tempest-VolumesBackupsTest-1817505426 None] Create backup finished. backup: e44184b1-0f9f-4e9c-a865-dd57d2a803d0. May 07 22:09:33.419920 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Restore backup started, backup: e44184b1-0f9f-4e9c-a865-dd57d2a803d0 volume: 9014cf04-9841-4a31-911d-9904ead789e7. May 07 22:09:33.447839 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:33.452509 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.487630 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.035s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:33.488135 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[834cb90d-a6b1-462a-93c1-5359a59a769a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:33.489632 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.516190 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[cbfc8742-1128-4365-840c-40b6d3db278d]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:33.517524 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:33.519311 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[cbfc8742-1128-4365-840c-40b6d3db278d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:33.521332 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.539255 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1fa80a5a-40ef-4e31-8631-f234b86cea12]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:33.539962 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:33.542103 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[1fa80a5a-40ef-4e31-8631-f234b86cea12]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:33.544661 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.585733 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.038s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:33.589427 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[462ac90d-8d3d-4b58-9d36-962ced65ce11]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:33.594957 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.633235 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.036s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:33.633595 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[d00ed081-4d2f-4797-be60-5e3e19f16d26]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:33.639082 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.655884 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:33.656241 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[3ff50f6f-b9dd-45a4-a925-4d6e16bbd320]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:33.658841 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:33.706353 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] CMD "nvme version" returned: 0 in 0.047s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:33.717620 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:09:33.718719 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:09:33.727439 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:09:33.728610 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] <== get_connector_properties: return (277ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:34.997203 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:09:35.000201 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9014cf04-9841-4a31-911d-9904ead789e7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9014cf04-9841-4a31-911d-9904ead789e7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:35.001530 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:09:35.114858 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] <== connect_volume: return (113ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:35.123648 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:09:35.154425 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Starting restore from Ceph backup=e44184b1-0f9f-4e9c-a865-dd57d2a803d0 to volume=volume-9014cf04-9841-4a31-911d-9904ead789e7 {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 07 22:09:35.292542 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Looking for snapshot of backup base 'volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:09:35.293212 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Found snapshot 'backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:09:35.311151 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Volume file is RBD. {{(pid=129863) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 07 22:09:35.401825 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Attempting differential restore. {{(pid=129863) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 07 22:09:35.402643 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Attempting incremental restore from base='volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' snap='backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526' {{(pid=129863) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 07 22:09:35.404657 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Performing differential transfer from 'volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' to 'volume-9014cf04-9841-4a31-911d-9904ead789e7' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:09:35.405234 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0@backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:09:35.406000 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_op6f1kud --pool volumes - volumes/volume-9014cf04-9841-4a31-911d-9904ead789e7' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:09:42.715147 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Restore transfer completed in 7.3099s {{(pid=129863) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 07 22:09:42.717055 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 07 22:09:42.751611 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Restoring volume base metadata {{(pid=129863) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 07 22:09:42.771167 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] No metadata of type 'volume-metadata' to restore {{(pid=129863) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 07 22:09:42.771549 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Restoring volume glance metadata {{(pid=129863) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 07 22:09:43.021387 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Restore to volume 9014cf04-9841-4a31-911d-9904ead789e7 finished successfully. {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 07 22:09:43.023317 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9014cf04-9841-4a31-911d-9904ead789e7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9014cf04-9841-4a31-911d-9904ead789e7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:43.039133 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] <== disconnect_volume: return (14ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:43.115585 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.manager [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Encryption key ID for volume 9014cf04-9841-4a31-911d-9904ead789e7 already matches encryption key ID in backup e44184b1-0f9f-4e9c-a865-dd57d2a803d0. {{(pid=129863) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 07 22:09:43.173981 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-34510e99-4c28-41e6-b03e-dbbc79248a29 tempest-VolumesBackupsTest-1817505426 None] Finished restoring backup e44184b1-0f9f-4e9c-a865-dd57d2a803d0 to volume 9014cf04-9841-4a31-911d-9904ead789e7. May 07 22:09:47.309245 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:09:47.310354 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:09:47.862760 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Delete backup started, backup: e44184b1-0f9f-4e9c-a865-dd57d2a803d0. May 07 22:09:47.866740 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Delete started for backup=e44184b1-0f9f-4e9c-a865-dd57d2a803d0 {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:09:47.869443 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Trying diff format basename='volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' for backup base image of volume ee967edb-ea50-423b-8d88-03897f067e75. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:09:48.089670 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Looking for snapshot of backup base 'volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:09:48.090060 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Found snapshot 'backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:09:48.090638 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Deleting backup snapshot='backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0.snap.1651961362.3965526' {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 07 22:09:48.830282 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Deleting backup base image='volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' of volume ee967edb-ea50-423b-8d88-03897f067e75. May 07 22:09:49.874417 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Base backup image='volume-ee967edb-ea50-423b-8d88-03897f067e75.backup.e44184b1-0f9f-4e9c-a865-dd57d2a803d0' of volume ee967edb-ea50-423b-8d88-03897f067e75 deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:09:49.962204 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Delete of backup 'e44184b1-0f9f-4e9c-a865-dd57d2a803d0' for volume 'ee967edb-ea50-423b-8d88-03897f067e75' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:09:49.970279 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:09:50.027430 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Created reservations ['ff18a19d-5a84-4e13-8681-f350a465e1da', 'fe4ea661-98a3-4ef0-a3d2-3ee86f4b04e9'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:09:50.081732 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-25d6e0af-9add-438e-8476-49d8b0a20cba tempest-VolumesBackupsTest-1817505426 None] Delete backup finished, backup e44184b1-0f9f-4e9c-a865-dd57d2a803d0 deleted. May 07 22:09:58.278966 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Create backup started, backup: bf1ec83c-ae3f-4463-8ab5-0a650aa070b0 volume: 2f787811-5f92-4037-af1b-c30ab8a36498. May 07 22:09:58.293809 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:09:58Z,data_timestamp=2022-05-07T22:09:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1738268498',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=bf1ec83c-ae3f-4463-8ab5-0a650aa070b0,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='270c72ab746a472d9bc29941c34f86e5',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='579aa8931a494de8bd0c337d020421d2',volume_id=2f787811-5f92-4037-af1b-c30ab8a36498) May 07 22:09:58.361544 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:58.369122 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.412758 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.049s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:58.413330 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[c13abf60-6df5-4ead-be7d-0701c28b7085]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:58.417731 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.488176 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ad27d1ab-f69d-4781-9750-1a7898b46859]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:58.488639 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:58.490637 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[ad27d1ab-f69d-4781-9750-1a7898b46859]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:58.493325 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.506848 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[39db1c34-7b40-467d-8433-ff7fadd0ca45]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:09:58.514718 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:09:58.518935 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[39db1c34-7b40-467d-8433-ff7fadd0ca45]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:58.523034 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.547303 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.023s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:58.547816 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[6c333cdb-ceaf-42c6-9440-8b9a338bc3a8]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:58.551025 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.580262 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.030s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:58.580680 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[7b740e68-0844-4786-ae4e-f5c7eaa2dd11]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:58.582809 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.619735 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.036s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:58.620422 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[46295c89-9529-41f9-95d1-8e11346d9493]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:09:58.621591 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:09:58.673892 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] CMD "nvme version" returned: 0 in 0.052s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:09:58.675837 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:09:58.676324 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:09:58.677674 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:09:58.678587 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] <== get_connector_properties: return (316ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:59.704830 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:09:59.706400 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2f787811-5f92-4037-af1b-c30ab8a36498', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2f787811-5f92-4037-af1b-c30ab8a36498', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:09:59.708105 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:09:59.806481 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] <== connect_volume: return (99ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:09:59.814230 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:09:59.851026 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Starting backup of volume='2f787811-5f92-4037-af1b-c30ab8a36498'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:09:59.856357 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Volume file is RBD: attempting optimized backup {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 07 22:09:59.919867 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Creating base image 'volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' {{(pid=129863) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 07 22:09:59.920504 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:10:00.015680 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Using --from-snap 'None' for incremental backup of volume 2f787811-5f92-4037-af1b-c30ab8a36498. {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 07 22:10:00.016388 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Creating backup snapshot='backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0.snap.1651961400.0159857' {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 07 22:10:00.613364 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Performing differential transfer from 'volume-2f787811-5f92-4037-af1b-c30ab8a36498' to 'volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:10:00.614070 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_p13hwuy5 --pool volumes volumes/volume-2f787811-5f92-4037-af1b-c30ab8a36498@backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0.snap.1651961400.0159857 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:10:00.614676 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:10:01.859783 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Differential backup transfer completed in 1.2455s {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 07 22:10:01.861440 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:10:01.912236 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:10:01.916631 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:10:01.933188 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:10:01.934854 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:10:01.942480 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] No metadata type 'volume-glance-metadata' available {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 07 22:10:01.943484 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Backing up metadata for volume 2f787811-5f92-4037-af1b-c30ab8a36498. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:10:02.011257 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Backup 'bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' of volume 2f787811-5f92-4037-af1b-c30ab8a36498 finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:10:02.013355 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2f787811-5f92-4037-af1b-c30ab8a36498', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2f787811-5f92-4037-af1b-c30ab8a36498', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:02.075944 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] <== disconnect_volume: return (61ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:02.263979 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-3e7dc908-41e3-4719-8be4-fd8085df7d99 tempest-VolumesBackupsV39Test-412393331 None] Create backup finished. backup: bf1ec83c-ae3f-4463-8ab5-0a650aa070b0. May 07 22:10:02.720693 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Delete backup started, backup: bf1ec83c-ae3f-4463-8ab5-0a650aa070b0. May 07 22:10:02.722335 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Delete started for backup=bf1ec83c-ae3f-4463-8ab5-0a650aa070b0 {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:10:02.723077 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Trying diff format basename='volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' for backup base image of volume 2f787811-5f92-4037-af1b-c30ab8a36498. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:10:02.852739 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Looking for snapshot of backup base 'volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:10:02.854135 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Found snapshot 'backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0.snap.1651961400.0159857' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:10:02.854725 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Deleting backup snapshot='backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0.snap.1651961400.0159857' {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 07 22:10:03.853536 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Deleting backup base image='volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' of volume 2f787811-5f92-4037-af1b-c30ab8a36498. May 07 22:10:04.299583 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Base backup image='volume-2f787811-5f92-4037-af1b-c30ab8a36498.backup.bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' of volume 2f787811-5f92-4037-af1b-c30ab8a36498 deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:10:04.382841 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Delete of backup 'bf1ec83c-ae3f-4463-8ab5-0a650aa070b0' for volume '2f787811-5f92-4037-af1b-c30ab8a36498' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:10:04.396440 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:10:04.466179 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Created reservations ['19565100-d51f-42f8-95ab-6fb34336efa0', '8cbbb2a0-d5a8-477e-bceb-2fde490fc88e'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:10:04.556977 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-b7af262a-8d43-46df-b909-012c75e18bcd tempest-VolumesBackupsV39Test-412393331 None] Delete backup finished, backup bf1ec83c-ae3f-4463-8ab5-0a650aa070b0 deleted. May 07 22:10:23.423896 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Create backup started, backup: 7835d3b6-ffbb-47df-80ed-ab232cb47e0a volume: 96b33d38-3791-4ff9-9b96-17fce364b1fb. May 07 22:10:23.442953 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-07T22:10:23Z,data_timestamp=2022-05-07T22:10:23Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=7835d3b6-ffbb-47df-80ed-ab232cb47e0a,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='d70782066cce4f2fbf5744b63e4195cb',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='4f4cbf51481d45de862752fe273dceda',volume_id=96b33d38-3791-4ff9-9b96-17fce364b1fb) May 07 22:10:23.515014 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:23.524564 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.538823 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.020s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:23.540566 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[b9eeca1a-fb1b-4bb1-a479-e221dd5eaa0a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:23.547464 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.564134 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[848b07b8-9546-4210-b78a-4835b087e9ff]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:10:23.564587 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:10:23.565997 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[848b07b8-9546-4210-b78a-4835b087e9ff]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:23.571229 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1cf2e761-e7b2-471d-bc50-292d0b311a39]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:10:23.589474 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:10:23.591005 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[1cf2e761-e7b2-471d-bc50-292d0b311a39]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:23.594660 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.624403 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.031s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:23.624874 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[3e8dc813-51b4-4b21-ab23-62dbc6c4ab21]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:23.626678 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.658840 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.028s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:23.659983 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[1e8c2ef1-e828-4302-a62c-4f1db4617f10]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:23.663695 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.698370 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.035s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:23.699023 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[16a622f2-5176-4ac8-a09f-15932894b294]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:23.700033 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:23.760455 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] CMD "nvme version" returned: 0 in 0.060s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:23.762196 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:10:23.762579 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:10:23.762980 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:10:23.763624 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] <== get_connector_properties: return (247ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:25.018064 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:10:25.021195 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-96b33d38-3791-4ff9-9b96-17fce364b1fb', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '96b33d38-3791-4ff9-9b96-17fce364b1fb', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:25.022502 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:10:25.141137 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] <== connect_volume: return (119ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:25.159360 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:10:25.202819 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Starting backup of volume='96b33d38-3791-4ff9-9b96-17fce364b1fb'. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 07 22:10:25.207247 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Volume file is RBD: attempting optimized backup {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 07 22:10:25.255584 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Creating base image 'volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' {{(pid=129863) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 07 22:10:25.256223 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129863) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 07 22:10:25.323695 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Using --from-snap 'None' for incremental backup of volume 96b33d38-3791-4ff9-9b96-17fce364b1fb. {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 07 22:10:25.324111 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Creating backup snapshot='backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725' {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 07 22:10:25.715246 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Performing differential transfer from 'volume-96b33d38-3791-4ff9-9b96-17fce364b1fb' to 'volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:10:25.716005 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_2q4dfri8 --pool volumes volumes/volume-96b33d38-3791-4ff9-9b96-17fce364b1fb@backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:10:25.716553 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:10:32.841450 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Differential backup transfer completed in 7.1259s {{(pid=129863) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 07 22:10:32.843500 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Getting metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 07 22:10:32.862157 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129863) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 07 22:10:32.862986 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Getting metadata type 'volume-metadata' {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 07 22:10:32.879544 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] No metadata type 'volume-metadata' available {{(pid=129863) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 07 22:10:32.880101 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Getting metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 07 22:10:32.889925 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129863) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 07 22:10:32.890764 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Backing up metadata for volume 96b33d38-3791-4ff9-9b96-17fce364b1fb. {{(pid=129863) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 07 22:10:32.951331 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Backup '7835d3b6-ffbb-47df-80ed-ab232cb47e0a' of volume 96b33d38-3791-4ff9-9b96-17fce364b1fb finished. {{(pid=129863) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 07 22:10:32.953081 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-96b33d38-3791-4ff9-9b96-17fce364b1fb', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '96b33d38-3791-4ff9-9b96-17fce364b1fb', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:32.995427 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] <== disconnect_volume: return (40ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:33.168921 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-afe7c794-194b-407d-bfd8-2397956a0f4f tempest-TestVolumeBackupRestore-1882080688 None] Create backup finished. backup: 7835d3b6-ffbb-47df-80ed-ab232cb47e0a. May 07 22:10:35.443923 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Restore backup started, backup: 7835d3b6-ffbb-47df-80ed-ab232cb47e0a volume: e4dd549f-c9a7-465c-a9bc-ac93b42c54fe. May 07 22:10:35.476059 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:35.481944 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.509901 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.029s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:35.510724 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[ba134d2a-4588-40e9-b652-275e5bd0679b]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c8574b80bd77\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:35.513075 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.529328 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[60114c0e-26c1-4b89-881f-3a8d7db81f1a]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:10:35.531147 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:10:35.532445 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[60114c0e-26c1-4b89-881f-3a8d7db81f1a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:35.535203 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.558020 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2bcaabd1-4f6b-4957-9c08-fbd562633aca]: [Errno 2] No such file or directory: 'systool' {{(pid=155525) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: Traceback (most recent call last): May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: ret = func(*f_args, **f_kwargs) May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: return func(*args, **kwargs) May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: return putils.execute(on_execute=on_execute, May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: obj = subprocess.Popen(cmd, May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: self._execute_child(args, executable, preexec_fn, close_fds, May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: raise child_exception_type(errno_num, err_msg, err_filename) May 07 22:10:35.558528 ncdv-master-1 cinder-backup[129863]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 07 22:10:35.559819 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[2bcaabd1-4f6b-4957-9c08-fbd562633aca]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:35.561700 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.595244 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.034s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:35.595690 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[dd233ae0-cc23-4fb6-9ea8-9a2e2a9c12c8]: (4, ('/dev/sda1\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:35.598732 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.626458 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.027s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:35.626873 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[9e72ff01-61aa-4305-98a6-b43aa0b537d0]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:35.629459 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.641850 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=155525) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:35.642474 ncdv-master-1 cinder-backup[129863]: DEBUG oslo.privsep.daemon [-] privsep: reply[a19d5834-476e-4d62-af36-421469822ec3]: (4, ('56a838d3-d241-3c40-8e35-eeab4e63734a\n', '')) {{(pid=155525) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 07 22:10:35.643268 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Running cmd (subprocess): nvme version {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 07 22:10:35.677093 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_concurrency.processutils [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] CMD "nvme version" returned: 0 in 0.033s {{(pid=129863) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 07 22:10:35.679694 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129863) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 07 22:10:35.680228 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 07 22:10:35.680785 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.lightos [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f dsc: {{(pid=129863) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 07 22:10:35.681782 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.utils [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] <== get_connector_properties: return (204ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c8574b80bd77', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '56a838d3-d241-3c40-8e35-eeab4e63734a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:302277cb-082b-4ae4-a74f-2c66af74892f', 'found_dsc': ''} {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:36.540300 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-45e49f68-24c9-47da-97eb-543f2ff8c192 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:10:36.758434 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connector [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Factory for rbd on None {{(pid=129863) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 07 22:10:36.760526 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e4dd549f-c9a7-465c-a9bc-ac93b42c54fe', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e4dd549f-c9a7-465c-a9bc-ac93b42c54fe', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:36.761686 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.linuxrbd [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] opening connection to ceph cluster (timeout=-1). {{(pid=129863) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 07 22:10:36.859354 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] <== connect_volume: return (98ms) {'path': } {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:36.872100 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Starting restore from Ceph backup=7835d3b6-ffbb-47df-80ed-ab232cb47e0a to volume=volume-e4dd549f-c9a7-465c-a9bc-ac93b42c54fe {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 07 22:10:37.008099 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Looking for snapshot of backup base 'volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:10:37.008867 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Found snapshot 'backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:10:37.015818 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Volume file is RBD. {{(pid=129863) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 07 22:10:37.078818 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Attempting differential restore. {{(pid=129863) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 07 22:10:37.079299 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Attempting incremental restore from base='volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' snap='backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725' {{(pid=129863) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 07 22:10:37.079663 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Performing differential transfer from 'volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' to 'volume-e4dd549f-c9a7-465c-a9bc-ac93b42c54fe' {{(pid=129863) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 07 22:10:37.080073 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a@backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725 -' into... {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 07 22:10:37.080465 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_cio_j9q7 --pool volumes - volumes/volume-e4dd549f-c9a7-465c-a9bc-ac93b42c54fe' {{(pid=129863) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 07 22:10:43.972347 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Restore transfer completed in 6.8920s {{(pid=129863) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 07 22:10:43.973444 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 07 22:10:44.025387 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Restoring volume base metadata {{(pid=129863) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 07 22:10:44.058518 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] No metadata of type 'volume-metadata' to restore {{(pid=129863) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 07 22:10:44.064566 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.driver [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Restoring volume glance metadata {{(pid=129863) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 07 22:10:44.357907 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Restore to volume e4dd549f-c9a7-465c-a9bc-ac93b42c54fe finished successfully. {{(pid=129863) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 07 22:10:44.361971 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e4dd549f-c9a7-465c-a9bc-ac93b42c54fe', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e4dd549f-c9a7-465c-a9bc-ac93b42c54fe', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 07 22:10:44.396816 ncdv-master-1 cinder-backup[129863]: DEBUG os_brick.initiator.connectors.rbd [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] <== disconnect_volume: return (33ms) None {{(pid=129863) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 07 22:10:44.509387 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.manager [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Encryption key ID for volume e4dd549f-c9a7-465c-a9bc-ac93b42c54fe already matches encryption key ID in backup 7835d3b6-ffbb-47df-80ed-ab232cb47e0a. {{(pid=129863) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 07 22:10:44.587345 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-544e87a9-e44d-4d0b-816e-fbeafd054123 tempest-TestVolumeBackupRestore-1882080688 None] Finished restoring backup 7835d3b6-ffbb-47df-80ed-ab232cb47e0a to volume e4dd549f-c9a7-465c-a9bc-ac93b42c54fe. May 07 22:10:47.305770 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:10:47.306903 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:11:47.303292 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:11:47.304078 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:12:29.448057 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Delete backup started, backup: 7835d3b6-ffbb-47df-80ed-ab232cb47e0a. May 07 22:12:29.458924 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Delete started for backup=7835d3b6-ffbb-47df-80ed-ab232cb47e0a {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 07 22:12:29.462035 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Trying diff format basename='volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' for backup base image of volume 96b33d38-3791-4ff9-9b96-17fce364b1fb. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 07 22:12:29.627729 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Looking for snapshot of backup base 'volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 07 22:12:29.629033 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Found snapshot 'backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725' {{(pid=129863) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 07 22:12:29.629837 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Deleting backup snapshot='backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a.snap.1651961425.323725' {{(pid=129863) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 07 22:12:30.021420 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Deleting backup base image='volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' of volume 96b33d38-3791-4ff9-9b96-17fce364b1fb. May 07 22:12:31.457822 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Base backup image='volume-96b33d38-3791-4ff9-9b96-17fce364b1fb.backup.7835d3b6-ffbb-47df-80ed-ab232cb47e0a' of volume 96b33d38-3791-4ff9-9b96-17fce364b1fb deleted. {{(pid=129863) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 07 22:12:31.562197 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.backup.drivers.ceph [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Delete of backup '7835d3b6-ffbb-47df-80ed-ab232cb47e0a' for volume '96b33d38-3791-4ff9-9b96-17fce364b1fb' finished. {{(pid=129863) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 07 22:12:31.569394 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_db.sqlalchemy.engines [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129863) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 07 22:12:31.654263 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.quota [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Created reservations ['9d75fd42-016f-4e6e-b138-71ad037a06e2', '9c0a511c-1a71-4c8b-8853-e00f7ae968ff'] {{(pid=129863) reserve /opt/stack/cinder/cinder/quota.py:755}} May 07 22:12:31.738303 ncdv-master-1 cinder-backup[129863]: INFO cinder.backup.manager [None req-ec5fb0d1-c380-4795-afde-ccab7924b667 tempest-TestVolumeBackupRestore-1882080688 None] Delete backup finished, backup 7835d3b6-ffbb-47df-80ed-ab232cb47e0a deleted. May 07 22:12:47.300128 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:12:47.300779 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:13:47.295522 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:13:47.296188 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:14:47.293474 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:14:47.294115 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:15:47.298667 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:15:47.300135 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:16:47.296749 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:16:47.297644 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:17:47.293592 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:17:47.294757 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:18:47.290544 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:18:47.291254 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:19:47.287703 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:19:47.288791 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:20:47.291127 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:20:47.291723 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:21:47.287392 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:21:47.287990 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:22:47.294020 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:22:47.294573 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:23:47.299069 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:23:47.299613 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:24:47.304697 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:24:47.305270 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:25:47.309253 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:25:47.309936 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:26:47.314792 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:26:47.315420 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:27:47.321182 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:27:47.321821 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:28:47.317877 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:28:47.318438 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:29:47.322634 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:29:47.323295 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:30:47.327916 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:30:47.328640 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:31:47.329963 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:31:47.330690 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:32:47.335462 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:32:47.336138 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:33:47.341104 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:33:47.341847 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:34:47.341798 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:34:47.342461 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:35:47.346411 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:35:47.347069 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:36:47.342948 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:36:47.343684 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:37:47.348064 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:37:47.348737 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:38:47.344643 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:38:47.345402 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:39:47.349304 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:39:47.349940 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:40:47.354928 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:40:47.355614 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:41:47.360509 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:41:47.361168 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:42:47.365286 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:42:47.365986 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:43:47.369721 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:43:47.370431 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:44:47.374483 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:44:47.375396 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:45:47.371507 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:45:47.372194 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:46:47.367341 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:46:47.368028 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:47:47.363524 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:47:47.364584 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:48:47.360056 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:48:47.360647 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:49:47.356473 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:49:47.357072 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:51:47.362088 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:51:47.362722 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:52:47.367797 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:52:47.368449 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:53:47.373461 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:53:47.374101 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:54:47.369769 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:54:47.370426 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:55:47.365992 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:55:47.366622 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:56:47.370628 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:56:47.371338 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:57:47.375911 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:57:47.376575 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:58:47.381452 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:58:47.382110 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 22:59:47.386648 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 22:59:47.387330 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 23:00:47.392027 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 23:00:47.392677 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 07 23:01:47.387510 ncdv-master-1 cinder-backup[129863]: DEBUG oslo_service.periodic_task [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129863) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 07 23:01:47.388162 ncdv-master-1 cinder-backup[129863]: DEBUG cinder.manager [None req-759466cb-bf32-4256-a447-fc4bb31644a4 None None] Notifying Schedulers of capabilities ... {{(pid=129863) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}