-- Logs begin at Sat 2022-04-09 20:01:14 UTC, end at Sat 2022-04-09 22:21:25 UTC. --
Apr 09 20:44:58.960282 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service.
Apr 09 20:45:02.402634 ncdv-master-1 cinder-backup[134469]: INFO cinder.cmd.backup [-] Backup running with 1 processes.
Apr 09 20:45:02.403622 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=134469) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}}
Apr 09 20:45:02.403910 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=134469) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}}
Apr 09 20:45:02.567464 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_db.sqlalchemy.engines [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134469) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 20:45:02.807161 ncdv-master-1 cinder-backup[134469]: INFO cinder.rpc [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Automatically selected cinder-backup objects version 1.39 as minimum service version.
Apr 09 20:45:02.811799 ncdv-master-1 cinder-backup[134469]: INFO cinder.rpc [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version.
Apr 09 20:45:02.816859 ncdv-master-1 cinder-backup[134469]: INFO cinder.rpc [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Automatically selected cinder-volume objects version 1.39 as minimum service version.
Apr 09 20:45:02.821283 ncdv-master-1 cinder-backup[134469]: INFO cinder.rpc [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version.
Apr 09 20:45:02.827445 ncdv-master-1 cinder-backup[134469]: INFO cinder.rpc [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version.
Apr 09 20:45:02.833228 ncdv-master-1 cinder-backup[134469]: INFO cinder.rpc [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version.
Apr 09 20:45:02.889236 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_db.sqlalchemy.engines [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134469) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 20:45:02.925450 ncdv-master-1 cinder-backup[134469]: INFO oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Starting 1 workers
Apr 09 20:45:02.930737 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Started child 134916 {{(pid=134469) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Apr 09 20:45:02.931946 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Full set of CONF: {{(pid=134469) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}}
Apr 09 20:45:02.932462 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] ******************************************************************************** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}}
Apr 09 20:45:02.932778 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] Configuration options gathered from: {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}}
Apr 09 20:45:02.933157 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}}
Apr 09 20:45:02.933754 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}}
Apr 09 20:45:02.934047 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] ================================================================================ {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}}
Apr 09 20:45:02.935166 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] allow_availability_zone_fallback = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.935563 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] allow_compression_on_image_upload = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.936063 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] allowed_direct_url_schemes     = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.936464 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] api_rate_limit                 = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.936948 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] auth_strategy                  = keystone {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.937372 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_availability_zone      = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.937762 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_api_class               = cinder.backup.api.API {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.938197 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_chunk_size         = 134217728 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.938734 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_conf               = /etc/ceph/ceph.conf {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.939121 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_image_journals     = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.939501 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_pool               = backups {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.940096 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_stripe_count       = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.940202 ncdv-master-1 cinder-backup[134916]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0)
Apr 09 20:45:02.940683 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_stripe_unit        = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.940980 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_ceph_user               = cinder-bak {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.941568 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_driver                  = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.942085 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_driver_init_check_interval = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.942575 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_driver_stats_polling_interval = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.942959 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_manager                 = cinder.backup.manager.BackupManager {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.943360 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_max_operations          = 15 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.943720 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_metadata_version        = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.944133 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_name_template           = backup-%s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.944619 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_native_threads_pool_size = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.945302 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_object_number_per_notification = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.945674 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_service_inithost_offload = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.946056 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_timer_interval          = 120 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.947048 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_use_same_host           = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.947430 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_use_temp_snapshot       = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.947875 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backup_workers                 = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.948292 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] chap_password                  = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.948830 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] chap_username                  =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.950428 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] chiscsi_conf                   = /etc/chelsio-iscsi/chiscsi.conf {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.950979 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_internal_tenant_project_id = a6cf2e406d07415cac661e73b93ccb02 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.951493 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_internal_tenant_user_id = 6504d578b1b64feb93ba69d947a971e9 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.952590 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] compression_format             = gzip {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.953050 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] config_dir                     = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.953360 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] config_file                    = ['/etc/cinder/cinder.conf'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.953754 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] config_source                  = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.954124 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] consistencygroup_api_class     = cinder.consistencygroup.api.API {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.954658 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] control_exchange               = openstack {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.955371 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] debug                          = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.955725 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] default_availability_zone      = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.956073 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] default_group_type             = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.956797 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.958426 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] default_volume_type            = ceph {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.958939 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] driver_client_cert             = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.959405 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] driver_client_cert_key         = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.959765 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] driver_data_namespace          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.960250 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] driver_ssl_cert_path           = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.960846 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] driver_ssl_cert_verify         = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.961233 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] driver_use_ssl                 = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.961727 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] enable_new_services            = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.962214 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] enable_unsupported_driver      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.962646 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] enabled_backends               = ['ceph'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.963025 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] enforce_multipath_for_image_xfer = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.963380 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] filter_function                = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.963730 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_api_insecure            = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.964099 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_api_servers             = ['http://10.222.0.30/image'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.964456 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_api_ssl_compression     = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.964937 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_ca_certificates_file    = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.965802 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_catalog_info            = image:glance:publicURL {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.966149 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_certfile                = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.966645 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_core_properties         = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.967023 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_keyfile                 = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.967430 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_num_retries             = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.967820 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] glance_request_timeout         = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.968228 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] goodness_function              = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.968656 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] graceful_shutdown_timeout      = 5 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.969224 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] group_api_class                = cinder.group.api.API {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.969593 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] host                           = ncdv-master-1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.969970 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iet_conf                       = /etc/iet/ietd.conf {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.970606 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_compress_on_upload       = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.971176 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_conversion_address_space_limit = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.971648 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_conversion_cpu_limit     = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.972297 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_conversion_dir           = /opt/stack/data/cinder/conversion {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.972658 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_upload_use_cinder_backend = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.973016 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_upload_use_internal_tenant = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.973385 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_volume_cache_enabled     = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.973741 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_volume_cache_max_count   = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.974110 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] image_volume_cache_max_size_gb = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.974477 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] instance_format                = [instance: %(uuid)s]  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.974966 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] instance_uuid_format           = [instance: %(uuid)s]  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.975356 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iscsi_iotype                   = fileio {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.975992 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iscsi_secondary_ip_addresses   = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.976410 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iscsi_target_flags             =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.976751 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iscsi_write_cache              = on {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.977164 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iser_helper                    = tgtadm {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.977759 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iser_ip_address                = 10.222.0.30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.978167 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iser_port                      = 3260 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.978539 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] iser_target_prefix             = iqn.2010-10.org.openstack: {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.978937 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_config_append              = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.979408 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_date_format                = %Y-%m-%d %H:%M:%S {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.979759 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_dir                        = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.980305 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_file                       = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.980809 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_options                    = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.981302 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_rotate_interval            = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.984309 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_rotate_interval_type       = days {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.985091 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] log_rotation_type              = none {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.985843 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] logging_context_format_string  = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.986700 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] logging_debug_format_suffix    = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.987269 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] logging_default_format_string  = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.987831 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] logging_exception_prefix       = ERROR %(name)s %(instance)s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.988312 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] logging_user_identity_format   = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.988880 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] max_age                        = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.989685 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] max_logfile_count              = 30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.990219 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] max_logfile_size_mb            = 200 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.991033 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] max_over_subscription_ratio    = 20.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.991692 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] message_reap_interval          = 86400 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.992351 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] message_ttl                    = 2592000 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.992898 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] monkey_patch                   = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.993471 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] monkey_patch_modules           = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.994052 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] my_ip                          = 10.222.0.30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.994613 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] no_snapshot_gb_quota           = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.995261 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] num_iser_scan_tries            = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.996058 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] num_shell_tries                = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.996598 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] num_volume_device_scan_tries   = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.997258 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] nvmet_ns_id                    = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.997830 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] nvmet_port_id                  = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.998403 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_max_limit                = 1000 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.998973 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_volume_ext_list          = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:02.999516 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.lockutils [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] Lock "cleanup_incomplete_backups_134469" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=134916) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}}
Apr 09 20:45:02.999844 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] Cleaning up incomplete backup operations.
Apr 09 20:45:03.000015 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_volume_extension         = ['cinder.api.contrib.standard_extensions'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.000563 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_volume_listen            = 0.0.0.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.001464 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_volume_listen_port       = 8776 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.002125 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_volume_use_ssl           = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.002743 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] osapi_volume_workers           = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.003299 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] per_volume_size_limit          = -1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.003978 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] periodic_fuzzy_delay           = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.004544 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] periodic_interval              = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.005753 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] project_id_regex               = [0-9a-f\-]+ {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.006561 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] public_endpoint                = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.007089 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] publish_errors                 = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.007644 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_backup_gigabytes         = 1000 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.008197 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_backups                  = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.008734 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_consistencygroups        = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.009402 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_driver                   = cinder.quota.DbQuotaDriver {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.009929 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_gigabytes                = 1000 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.010478 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_groups                   = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.011240 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_snapshots                = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.011896 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] quota_volumes                  = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.012562 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] rate_limit_burst               = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.013125 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] rate_limit_except_level        = CRITICAL {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.013652 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] rate_limit_interval            = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.014314 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] replication_device             = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.014831 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] report_discard_supported       = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.015358 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] report_interval                = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.016046 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] reservation_clean_interval     = 86400 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.016746 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] reservation_expire             = 86400 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.017295 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] reserved_percentage            = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.017878 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] resource_query_filters_file    = /etc/cinder/resource_filters.json {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.018702 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] restore_discard_excess_bytes   = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.019247 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] rootwrap_config                = /etc/cinder/rootwrap.conf {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.019919 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] rpc_response_timeout           = 120 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.020463 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] run_external_periodic_tasks    = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.021004 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] scheduler_manager              = cinder.scheduler.manager.SchedulerManager {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.021886 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] scst_target_driver             = iscsi {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.022531 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] scst_target_iqn_name           = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.023007 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_down_time              = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.023541 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] snapshot_name_template         = snapshot-%s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.024195 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] split_loggers                  = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.024797 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] state_path                     = /opt/stack/data/cinder {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.025471 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] storage_availability_zone      = nova {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.026116 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] storage_protocol               = iscsi {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.027112 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] syslog_log_facility            = LOG_USER {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.027677 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] target_helper                  = lioadm {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.028298 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] target_ip_address              = 10.222.0.30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.029341 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] target_port                    = 3260 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.032023 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] target_prefix                  = iqn.2010-10.org.openstack: {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.032805 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] target_protocol                = iscsi {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.033520 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] trace_flags                    = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.034088 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] transfer_api_class             = cinder.transfer.api.API {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.036274 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] transport_url                  = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.037143 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] until_refresh                  = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.037693 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_chap_auth                  = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.038235 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_default_quota_class        = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.038788 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_eventlog                   = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.039326 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_journal                    = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.039961 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_json                       = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.040541 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_multipath_for_image_xfer   = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.041103 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_stderr                     = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.041954 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] use_syslog                     = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.042647 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] verify_glance_signatures       = enabled {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.043179 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_api_class               = cinder.volume.api.API {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.043724 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_backend_name            = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.044301 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_clear                   = zero {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.044928 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_clear_ionice            = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.045481 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_clear_size              = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.046277 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_copy_blkio_cgroup_name  = cinder-volume-copy {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.046998 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_copy_bps_limit          = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.047673 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_dd_blocksize            = 1M {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.048260 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_manager                 = cinder.volume.manager.VolumeManager {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.048838 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_name_template           = volume-%s {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.049444 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volume_usage_audit_period      = month {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.050188 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] volumes_dir                    = /opt/stack/data/cinder/volumes {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.050748 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] watch_log_file                 = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Apr 09 20:45:03.051461 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.052352 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.053087 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.enforce_new_defaults = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.053671 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.enforce_scope      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.054290 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.policy_default_rule = default {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.055009 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.policy_dirs        = ['policy.d'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.055616 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.policy_file        = policy.yaml {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.056448 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.057234 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.057910 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.058460 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.059062 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.059676 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_concurrency.disable_process_locking = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.060359 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_concurrency.lock_path     = /opt/stack/data/cinder {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.060995 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.admin_password = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.061601 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.admin_tenant_name = admin {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.062411 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.admin_token = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.063071 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.admin_user  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.063665 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_admin_prefix =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.064230 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_host   = 127.0.0.1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.064953 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_port   = 35357 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.065550 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_protocol = https {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.066310 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_section = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.066878 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_type   = password {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.067535 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_uri    = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.068195 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.auth_version = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.068735 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.cache       = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.069338 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.cafile      = /opt/stack/data/ca-bundle.pem {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.069880 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.certfile    = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.070499 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.delay_auth_decision = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.071062 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.071591 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.http_connect_timeout = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.072492 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.073350 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.identity_uri = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.073898 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.include_service_catalog = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.074445 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.insecure    = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.075135 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.interface   = public {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.075690 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.keyfile     = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.076275 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.076913 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.077676 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.078387 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.078926 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.079480 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_secret_key = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.080036 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_security_strategy = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.080728 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.081353 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.081885 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.region_name = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.082836 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.083528 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.service_token_roles_required = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.084218 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.service_type = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.084783 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.token_cache_time = 300 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.085504 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] keystone_authtoken.www_authenticate_uri = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.086119 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.backend                  = dogpile.cache.null {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.086811 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.backend_argument         = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.087343 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.config_prefix            = cache.oslo {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.088234 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.dead_timeout             = 60.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.088750 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.debug_cache_backend      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.089276 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.enable_retry_client      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.090146 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.enable_socket_keepalive  = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.091018 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.enabled                  = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.091712 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.expiration_time          = 600 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.092212 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.hashclient_retry_attempts = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.093392 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.hashclient_retry_delay   = 1.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.093948 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_dead_retry      = 300 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.094646 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.095175 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.095809 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_pool_maxsize    = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.096366 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_pool_unused_timeout = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.097069 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_servers         = ['localhost:11211'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.098017 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.memcache_socket_timeout  = 1.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.098771 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.proxies                  = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.099272 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.retry_attempts           = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.099791 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.retry_delay              = 0.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.100348 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.socket_keepalive_count   = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.101050 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.socket_keepalive_idle    = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.101556 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.socket_keepalive_interval = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.102103 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.tls_allowed_ciphers      = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.103339 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.tls_cafile               = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.104175 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.tls_certfile             = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.104874 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.tls_enabled              = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.105682 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cache.tls_keyfile              = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.106277 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] coordination.backend_url       = etcd3+http://10.222.0.30:2379 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.107240 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.backend               = sqlalchemy {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.107746 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.connection            = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.108487 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.connection_debug      = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.109224 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.connection_parameters =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.109735 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.connection_recycle_time = 3600 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.110313 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.connection_trace      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.110954 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.db_inc_retry_interval = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.111860 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.db_max_retries        = 20 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.112448 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.db_max_retry_interval = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.112966 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.db_retry_interval     = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.113901 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.max_overflow          = 50 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.114421 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.max_pool_size         = 5 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.115066 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.max_retries           = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.115591 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.mysql_enable_ndb      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.116187 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.mysql_sql_mode        = TRADITIONAL {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.116685 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.pool_timeout          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.117328 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.retry_interval        = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.117843 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.slave_connection      = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.118497 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.sqlite_synchronous    = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.119212 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] database.use_db_reconnect      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.119758 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.120325 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.127086 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_metrics.metrics_process_name =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.128066 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.128959 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.129725 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.connection_string     = messaging:// {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.130291 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.enabled               = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.130812 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.es_doc_type           = notification {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.131322 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.es_scroll_size        = 10000 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.131840 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.es_scroll_time        = 2m {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.132363 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.filter_error_trace    = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.132892 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.hmac_keys             = SECRET_KEY {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.133442 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.sentinel_service_name = mymaster {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.134127 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.socket_timeout        = 0.1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.134757 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] profiler.trace_sqlalchemy      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.135335 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] privsep_osbrick.capabilities   = [21] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.136343 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] privsep_osbrick.group          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.136845 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] privsep_osbrick.helper_command = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.137488 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] privsep_osbrick.logger_name    = oslo_privsep.daemon {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.137993 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] privsep_osbrick.thread_pool_size = 4 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.138609 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] privsep_osbrick.user           = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.139169 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_sys_admin.capabilities  = [0, 1, 2, 3, 12, 21] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.139768 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_sys_admin.group         = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.140260 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_sys_admin.helper_command = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.140764 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_sys_admin.logger_name   = oslo_privsep.daemon {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.141306 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.141816 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] cinder_sys_admin.user          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.142336 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.auth_section      = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.143011 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.auth_type         = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.143517 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.cafile            = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.144105 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.certfile          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.144638 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.collect_timing    = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.145291 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.insecure          = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.145780 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.keyfile           = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.146721 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.send_service_user_token = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.147260 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.split_loggers     = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.147856 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] service_user.timeout           = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.148421 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.backend_availability_zone = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.149088 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.chap_password = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.149808 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.chap_username =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.150320 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.chiscsi_conf  = /etc/chelsio-iscsi/chiscsi.conf {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.150840 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.151372 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.151901 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.driver_client_cert = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.152405 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.driver_client_cert_key = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.152900 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.driver_data_namespace = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.153448 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.driver_ssl_cert_path = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.153969 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.154548 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.driver_use_ssl = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.155050 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.enable_deferred_deletion = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.155547 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.enable_unsupported_driver = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.156068 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.156940 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.filter_function = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.157463 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.goodness_function = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.158184 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iet_conf      = /etc/iet/ietd.conf {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.158866 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.159379 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.160009 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.image_volume_cache_enabled = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.160519 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.161103 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.161638 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iscsi_iotype  = fileio {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.162144 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.162671 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iscsi_target_flags =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.163200 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iscsi_write_cache = on {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.163732 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iser_helper   = tgtadm {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.164505 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iser_ip_address = 10.222.0.30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.165004 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iser_port     = 3260 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.165527 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.166076 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.166969 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.167477 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.num_shell_tries = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.168079 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.168590 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.nvmet_ns_id   = 10 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.169247 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.nvmet_port_id = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.169880 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rados_connect_timeout = -1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.170392 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rados_connection_interval = 5 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.170916 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rados_connection_retries = 3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.171447 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_ceph_conf =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.171953 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_cluster_name = ceph {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.172557 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.173091 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.173604 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.174146 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_pool      = rbd {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.174727 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_secret_uuid = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.175249 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.175739 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.rbd_user      = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.176257 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.replication_connect_timeout = 5 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.177156 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.replication_device = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.177677 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.report_discard_supported = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.178265 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.178795 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.reserved_percentage = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.179437 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.scst_target_driver = iscsi {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.180038 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.scst_target_iqn_name = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.180550 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.storage_protocol = iscsi {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.181076 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.target_helper = tgtadm {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.181701 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.target_ip_address = 10.222.0.30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.182204 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.target_port   = 3260 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.182719 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.183227 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.target_protocol = iscsi {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.183714 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.trace_flags   = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.184231 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.unique_fqdn_network = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.184823 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.use_chap_auth = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.187317 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.187856 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_backend_name = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.188548 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_clear  = zero {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.189096 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_clear_ionice = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.189968 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_clear_size = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.190588 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.191103 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.191640 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.192269 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] backend_defaults.volumes_dir   = /opt/stack/data/cinder/volumes {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.192815 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_reports.file_event_handler = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.193514 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_reports.file_event_handler_interval = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.194044 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_reports.log_dir           = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.194723 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.195237 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.195769 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.196269 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.197222 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.197724 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.198321 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.198819 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.199361 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.199958 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.200622 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.201147 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.201682 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.202206 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.202800 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.203326 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.203900 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.204411 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.205018 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.205662 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.206182 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.206831 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.207712 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.ssl      = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.208248 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.ssl_ca_file =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.208855 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.ssl_cert_file =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.209402 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.ssl_key_file =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.210041 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_rabbit.ssl_version =  {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.210724 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_notifications.driver = [] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.211248 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_notifications.retry = -1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.211828 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.212328 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] oslo_messaging_notifications.transport_url = **** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.212913 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] key_manager.backend            = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.213502 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.auth_endpoint         = http://localhost/identity/v3 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.214075 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.barbican_api_version  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.214610 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.barbican_endpoint     = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.215242 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.barbican_endpoint_type = public {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.215735 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.barbican_region_name  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.216223 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.cafile                = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.216714 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.certfile              = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.217729 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.collect_timing        = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.218231 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.insecure              = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.218727 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.keyfile               = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.219228 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.number_of_retries     = 60 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.219744 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.retry_delay           = 1 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.220347 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.send_service_user_token = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.221017 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.split_loggers         = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.221591 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.timeout               = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.222103 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.verify_ssl            = True {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.222604 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican.verify_ssl_path       = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.223120 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.auth_section = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.223633 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.auth_type = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.224209 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.cafile   = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.224699 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.certfile = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.225320 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.collect_timing = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.225813 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.insecure = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.226498 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.keyfile  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.227636 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.split_loggers = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.228369 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] barbican_service_user.timeout  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.229027 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.approle_role_id          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.229571 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.approle_secret_id        = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.230194 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.cafile                   = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.230703 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.certfile                 = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.231396 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.collect_timing           = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.231902 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.insecure                 = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.232479 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.keyfile                  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.233079 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.kv_mountpoint            = secret {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.233588 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.kv_version               = 2 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.234508 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.namespace                = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.235017 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.root_token_id            = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.235588 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.split_loggers            = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.236389 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.ssl_ca_crt_file          = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.236929 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.timeout                  = None {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.237844 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.use_ssl                  = False {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.238362 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] vault.vault_url                = http://127.0.0.1:8200 {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Apr 09 20:45:03.238797 ncdv-master-1 cinder-backup[134469]: DEBUG oslo_service.service [None req-7cba246f-5c75-48f7-aede-3613918c43ca None None] ******************************************************************************** {{(pid=134469) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}}
Apr 09 20:45:03.256544 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 20:45:03.281345 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.lockutils [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] Lock "cleanup_incomplete_backups_134469" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.282s {{(pid=134916) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}}
Apr 09 20:45:03.288725 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 20:45:03.295766 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:45:03.340921 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.service [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] Creating RPC server for service cinder-backup {{(pid=134916) start /opt/stack/cinder/cinder/service.py:231}}
Apr 09 20:45:03.358741 ncdv-master-1 cinder-backup[134916]: INFO cinder.keymgr.migration [None req-b30b84a6-9246-4d1b-bdf7-086cf6e87c86 None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Apr 09 20:45:03.373105 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.service [None req-fdc82d7e-98ee-425b-9da4-ecb8b78d7643 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=134916) start /opt/stack/cinder/cinder/service.py:237}}
Apr 09 20:46:38.400932 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:46:38.401747 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:47:38.396980 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:47:38.397814 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:48:38.393549 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:48:38.394979 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:49:38.391803 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:49:38.392941 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:50:38.390965 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:50:38.392087 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:51:38.390373 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:51:38.391661 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:52:38.389201 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:52:38.390375 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:53:38.389302 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:53:38.390689 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:54:38.388972 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:54:38.389636 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:55:38.394649 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:55:38.395359 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:56:38.400170 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:56:38.400781 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:57:38.401814 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:57:38.402449 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:58:38.398153 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:58:38.398815 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 20:59:38.403880 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 20:59:38.404588 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:00:38.409133 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:00:38.409875 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:01:38.414689 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:01:38.416322 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:02:38.418926 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:02:38.419573 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:03:38.422303 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:03:38.423157 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:04:38.428196 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:04:38.428818 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:05:38.424413 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:05:38.425946 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:06:38.422937 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:06:38.423611 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:07:38.422092 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:07:38.422723 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:08:38.419334 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:08:38.420644 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:09:38.425508 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:09:38.427486 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:10:38.421754 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:10:38.422355 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:11:38.428638 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:11:38.429284 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:12:38.426439 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:12:38.427316 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:12:48.614516 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Create backup started, backup: b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6 volume: c5d2f8ab-7872-4061-be6c-5b1930fc0017.
Apr 09 21:12:48.650339 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:12:48Z,data_timestamp=2022-04-09T21:12:48Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-231773288',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='9bd28da6b23940dc816d678fc6f2b9db',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='74311b56951549df8388565bff4a5499',volume_id=c5d2f8ab-7872-4061-be6c-5b1930fc0017)
Apr 09 21:12:48.796745 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:12:48.798721 ncdv-master-1 cinder-backup[134916]: INFO oslo.privsep.daemon [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp9xfxpsls/privsep.sock']
Apr 09 21:12:48.997185 ncdv-master-1 sudo[159528]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp9xfxpsls/privsep.sock
Apr 09 21:12:49.010697 ncdv-master-1 sudo[159528]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 09 21:12:53.644912 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-3f6514b9-f05a-423b-9d37-299800de2387 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:12:54.689576 ncdv-master-1 sudo[159528]: pam_unix(sudo:session): session closed for user root
Apr 09 21:12:54.702631 ncdv-master-1 cinder-backup[134916]: INFO oslo.privsep.daemon [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Spawned new privsep daemon via rootwrap
Apr 09 21:12:54.703211 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Accepted privsep connection to /tmp/tmp9xfxpsls/privsep.sock {{(pid=134916) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}}
Apr 09 21:12:54.704778 ncdv-master-1 cinder-backup[134916]: INFO oslo.privsep.daemon [-] privsep daemon starting
Apr 09 21:12:54.705022 ncdv-master-1 cinder-backup[134916]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
Apr 09 21:12:54.705340 ncdv-master-1 cinder-backup[134916]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
Apr 09 21:12:54.705640 ncdv-master-1 cinder-backup[134916]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 159908
Apr 09 21:12:54.711071 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[96335f2e-4754-44b9-b792-86488ab06432]: (2,) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.172932 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.236309 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.064s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:12:55.238069 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[2dccd436-d880-43fe-bdbb-e8314d33a39e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.240354 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.267387 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5ea2695a-5225-4a7a-8814-8c98be5da69c]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:12:55.301714 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:12:55.304495 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[5ea2695a-5225-4a7a-8814-8c98be5da69c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.311247 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.329352 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[8ce5d488-0641-4440-8630-55150f2f8cdd]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:12:55.330456 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:12:55.331900 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[8ce5d488-0641-4440-8630-55150f2f8cdd]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.341610 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.396530 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.052s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:12:55.396990 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[76c6b115-42bf-4350-92e7-89d5e82f4a81]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.399016 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.495904 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.096s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:12:55.496242 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[85a6123e-ecf8-45ae-864a-010be4538ec9]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.497685 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.511515 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:12:55.513204 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[c359f58e-d8e6-4ac0-8235-b0a2346f2bf8]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:12:55.515292 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:12:55.554960 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] CMD "nvme version" returned: 0 in 0.040s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:12:55.585005 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:12:55.585659 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:12:55.586201 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:12:55.587961 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] <== get_connector_properties: return (6790ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:12:56.533164 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:12:56.536228 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee3e8d90>, 'connection_properties': {'name': 'volumes/volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'c5d2f8ab-7872-4061-be6c-5b1930fc0017', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:12:56.539438 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:12:56.657627 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] <== connect_volume: return (120ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ee23c2e0>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:12:56.702772 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Starting backup of volume='c5d2f8ab-7872-4061-be6c-5b1930fc0017'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:12:56.703660 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Volume file is RBD: attempting optimized backup {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}}
Apr 09 21:12:56.764235 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Creating base image 'volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' {{(pid=134916) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}}
Apr 09 21:12:56.764695 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:12:58.028662 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Using --from-snap 'None' for incremental backup of volume c5d2f8ab-7872-4061-be6c-5b1930fc0017. {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}}
Apr 09 21:12:58.029428 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Creating backup snapshot='backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6.snap.1649538778.028683' {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}}
Apr 09 21:12:59.015576 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Performing differential transfer from 'volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017' to 'volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:12:59.015997 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_jai8ygr1 --pool volumes volumes/volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017@backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6.snap.1649538778.028683 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:12:59.016376 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:13:00.109375 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Differential backup transfer completed in 1.0936s {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}}
Apr 09 21:13:00.111150 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:13:00.134343 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:13:00.134734 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:13:00.158655 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:13:00.159142 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:13:00.171455 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] No metadata type 'volume-glance-metadata' available {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Apr 09 21:13:00.172153 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Backing up metadata for volume c5d2f8ab-7872-4061-be6c-5b1930fc0017. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:13:00.238773 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Backup 'b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' of volume c5d2f8ab-7872-4061-be6c-5b1930fc0017 finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:13:00.240491 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee3e8d90>, 'connection_properties': {'name': 'volumes/volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'c5d2f8ab-7872-4061-be6c-5b1930fc0017', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ee23c2e0>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:00.255814 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] <== disconnect_volume: return (14ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:00.388710 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-b11ae8eb-0c1d-49f5-88fb-bb45efc8813a tempest-VolumesBackupsTest-1881101193 None] Create backup finished. backup: b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6.
Apr 09 21:13:01.144374 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Delete backup started, backup: b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6.
Apr 09 21:13:01.146591 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Delete started for backup=b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6 {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:13:01.147163 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Trying diff format basename='volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' for backup base image of volume c5d2f8ab-7872-4061-be6c-5b1930fc0017. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:13:01.236288 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Looking for snapshot of backup base 'volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:13:01.236694 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Found snapshot 'backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6.snap.1649538778.028683' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:13:01.237782 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Deleting backup snapshot='backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6.snap.1649538778.028683' {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}}
Apr 09 21:13:02.244498 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Deleting backup base image='volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' of volume c5d2f8ab-7872-4061-be6c-5b1930fc0017.
Apr 09 21:13:02.554354 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Base backup image='volume-c5d2f8ab-7872-4061-be6c-5b1930fc0017.backup.b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' of volume c5d2f8ab-7872-4061-be6c-5b1930fc0017 deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:13:02.637308 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Delete of backup 'b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6' for volume 'c5d2f8ab-7872-4061-be6c-5b1930fc0017' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:13:02.644484 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:13:02.708545 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Created reservations ['0e106f85-b46c-4759-87a7-1f180129ac6d', 'add0adc2-f251-4b9b-b1ab-dac599e03186'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:13:02.763828 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-711a2f69-b8ac-47f0-a13f-4312d890a4b0 tempest-VolumesBackupsTest-1881101193 None] Delete backup finished, backup b0ffc5f3-dc5c-4e8d-afcd-3f0084df16c6 deleted.
Apr 09 21:13:16.052602 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Create backup started, backup: 2a613956-d142-4205-b578-5cec6949c9c2 volume: 7fc9428d-ac71-4320-a6b2-4e0c1283dd81.
Apr 09 21:13:16.080983 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:13:15Z,data_timestamp=2022-04-09T21:13:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1530849055',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=2a613956-d142-4205-b578-5cec6949c9c2,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='9bd28da6b23940dc816d678fc6f2b9db',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='74311b56951549df8388565bff4a5499',volume_id=7fc9428d-ac71-4320-a6b2-4e0c1283dd81)
Apr 09 21:13:16.145288 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:16.149243 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.179935 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.032s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:16.180412 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[356a4cd6-d916-4783-a1b9-95d555e9a5f3]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:16.182191 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.216137 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[544cb945-f84e-4a2c-8861-2b56aa6e1d72]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:13:16.217671 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:13:16.219315 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[544cb945-f84e-4a2c-8861-2b56aa6e1d72]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:16.222331 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.246131 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a939b85b-dcfb-4f62-b53a-d008ddf7a0aa]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:13:16.246543 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:13:16.248004 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[a939b85b-dcfb-4f62-b53a-d008ddf7a0aa]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:16.250371 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.290707 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.040s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:16.291186 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[c7e2ea99-5da6-4ab0-b2ca-27b3074884a3]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:16.293222 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.338012 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.045s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:16.338747 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[6f15a351-c52c-48b0-b0c5-fc4cc8b2529e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:16.340247 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.390651 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.050s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:16.391764 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[91f01a33-7cda-4a72-84d0-fdf9f0e6fc65]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:16.392770 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:16.469887 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] CMD "nvme version" returned: 0 in 0.077s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:16.471897 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:13:16.472640 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:13:16.473204 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:13:16.474637 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] <== get_connector_properties: return (328ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:17.821138 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:13:17.822602 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee246970>, 'connection_properties': {'name': 'volumes/volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7fc9428d-ac71-4320-a6b2-4e0c1283dd81', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:17.823653 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:13:17.957769 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] <== connect_volume: return (134ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ee176b80>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:17.966490 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:13:18.026493 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Starting backup of volume='7fc9428d-ac71-4320-a6b2-4e0c1283dd81'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:13:18.029250 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Volume file is RBD: attempting optimized backup {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}}
Apr 09 21:13:18.091454 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Creating base image 'volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' {{(pid=134916) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}}
Apr 09 21:13:18.091942 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:13:18.159717 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Using --from-snap 'None' for incremental backup of volume 7fc9428d-ac71-4320-a6b2-4e0c1283dd81. {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}}
Apr 09 21:13:18.160585 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Creating backup snapshot='backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974' {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}}
Apr 09 21:13:18.669085 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Performing differential transfer from 'volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81' to 'volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:13:18.670000 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_xnqi7js8 --pool volumes volumes/volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81@backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:13:18.670658 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:13:28.484567 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Differential backup transfer completed in 9.8157s {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}}
Apr 09 21:13:28.485336 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:13:28.502051 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:13:28.504340 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:13:28.521367 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:13:28.521752 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:13:28.539870 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}}
Apr 09 21:13:28.541660 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Backing up metadata for volume 7fc9428d-ac71-4320-a6b2-4e0c1283dd81. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:13:28.640349 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Backup '2a613956-d142-4205-b578-5cec6949c9c2' of volume 7fc9428d-ac71-4320-a6b2-4e0c1283dd81 finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:13:28.642803 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee246970>, 'connection_properties': {'name': 'volumes/volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7fc9428d-ac71-4320-a6b2-4e0c1283dd81', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ee176b80>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:28.692284 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] <== disconnect_volume: return (48ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:28.793707 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-053e0114-efdb-45ba-b450-81ce989c98f5 tempest-VolumesBackupsTest-1881101193 None] Create backup finished. backup: 2a613956-d142-4205-b578-5cec6949c9c2.
Apr 09 21:13:31.181077 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Restore backup started, backup: 2a613956-d142-4205-b578-5cec6949c9c2 volume: 42ab02bf-5053-4db7-a4de-c9a00ccd3537.
Apr 09 21:13:31.219472 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:31.222607 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.239781 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.017s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:31.240133 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[cb89026c-cde7-476a-a8db-dabd2af8c269]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:31.244122 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.251462 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5e92c34f-8fd0-4cf9-b4f1-1884e6143429]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:13:31.251996 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:13:31.253885 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[5e92c34f-8fd0-4cf9-b4f1-1884e6143429]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:31.256101 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.303485 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[aa7acc2f-b50c-49e4-be2e-3578c91b0b2e]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:13:31.303790 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:13:31.304970 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[aa7acc2f-b50c-49e4-be2e-3578c91b0b2e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:31.308457 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.348591 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.042s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:31.349352 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[2c9c5fa1-45aa-4527-a350-1878f00751aa]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:31.353219 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.416490 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.065s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:31.416778 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[064616ec-dbda-4485-a021-d343393df5ad]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:31.424210 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.446091 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.024s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:31.446646 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[dbf939c6-9a21-400e-8897-e1680305cea0]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:13:31.448443 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:13:31.513026 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] CMD "nvme version" returned: 0 in 0.064s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:13:31.514864 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:13:31.515260 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:13:31.515635 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:13:31.516522 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] <== get_connector_properties: return (295ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:32.853542 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:13:32.855772 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed892160>, 'connection_properties': {'name': 'volumes/volume-42ab02bf-5053-4db7-a4de-c9a00ccd3537', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '42ab02bf-5053-4db7-a4de-c9a00ccd3537', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:32.861499 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:13:32.998114 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] <== connect_volume: return (141ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed882a90>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:33.006346 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:13:33.026733 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Starting restore from Ceph backup=2a613956-d142-4205-b578-5cec6949c9c2 to volume=volume-42ab02bf-5053-4db7-a4de-c9a00ccd3537 {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}}
Apr 09 21:13:33.142930 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Looking for snapshot of backup base 'volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:13:33.143613 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Found snapshot 'backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:13:33.149672 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Volume file is RBD. {{(pid=134916) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}}
Apr 09 21:13:33.219254 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Attempting differential restore. {{(pid=134916) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}}
Apr 09 21:13:33.220254 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Attempting incremental restore from base='volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' snap='backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974' {{(pid=134916) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}}
Apr 09 21:13:33.220896 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Performing differential transfer from 'volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' to 'volume-42ab02bf-5053-4db7-a4de-c9a00ccd3537' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:13:33.221578 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2@backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:13:33.222185 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_x251ik_b --pool volumes - volumes/volume-42ab02bf-5053-4db7-a4de-c9a00ccd3537' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:13:38.424493 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:13:38.425256 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:13:41.602278 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Restore transfer completed in 8.3811s {{(pid=134916) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}}
Apr 09 21:13:41.603356 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}}
Apr 09 21:13:41.684571 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Restoring volume base metadata {{(pid=134916) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Apr 09 21:13:41.718344 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] No metadata of type 'volume-metadata' to restore {{(pid=134916) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Apr 09 21:13:41.718752 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Restoring volume glance metadata {{(pid=134916) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}}
Apr 09 21:13:42.046453 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Restore to volume 42ab02bf-5053-4db7-a4de-c9a00ccd3537 finished successfully. {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}}
Apr 09 21:13:42.048450 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed892160>, 'connection_properties': {'name': 'volumes/volume-42ab02bf-5053-4db7-a4de-c9a00ccd3537', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '42ab02bf-5053-4db7-a4de-c9a00ccd3537', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed882a90>}, 'force': True, ': False}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:13:42.073738 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] <== disconnect_volume: return (24ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:13:42.145690 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.manager [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Encryption key ID for volume 42ab02bf-5053-4db7-a4de-c9a00ccd3537 already matches encryption key ID in backup 2a613956-d142-4205-b578-5cec6949c9c2. {{(pid=134916) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}}
Apr 09 21:13:42.207042 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-277d8d57-c1d6-4a88-9050-fb3b303bc8ff tempest-VolumesBackupsTest-1881101193 None] Finished restoring backup 2a613956-d142-4205-b578-5cec6949c9c2 to volume 42ab02bf-5053-4db7-a4de-c9a00ccd3537.
Apr 09 21:13:46.560936 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Delete backup started, backup: 2a613956-d142-4205-b578-5cec6949c9c2.
Apr 09 21:13:46.562818 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Delete started for backup=2a613956-d142-4205-b578-5cec6949c9c2 {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:13:46.563381 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Trying diff format basename='volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' for backup base image of volume 7fc9428d-ac71-4320-a6b2-4e0c1283dd81. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:13:46.669167 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Looking for snapshot of backup base 'volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:13:46.669568 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Found snapshot 'backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:13:46.670011 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Deleting backup snapshot='backup.2a613956-d142-4205-b578-5cec6949c9c2.snap.1649538798.15974' {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}}
Apr 09 21:13:47.608254 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Deleting backup base image='volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' of volume 7fc9428d-ac71-4320-a6b2-4e0c1283dd81.
Apr 09 21:13:48.719206 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Base backup image='volume-7fc9428d-ac71-4320-a6b2-4e0c1283dd81.backup.2a613956-d142-4205-b578-5cec6949c9c2' of volume 7fc9428d-ac71-4320-a6b2-4e0c1283dd81 deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:13:48.810092 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Delete of backup '2a613956-d142-4205-b578-5cec6949c9c2' for volume '7fc9428d-ac71-4320-a6b2-4e0c1283dd81' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:13:48.818808 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:13:48.855612 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Created reservations ['19c8e08f-44f8-4596-b743-c2459409aa2e', '9ca30357-2566-411a-bf09-eb64c1215441'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:13:48.907223 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-2442ddec-649c-46fd-89bc-cb254b3f641b tempest-VolumesBackupsTest-1881101193 None] Delete backup finished, backup 2a613956-d142-4205-b578-5cec6949c9c2 deleted.
Apr 09 21:14:38.423457 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:14:38.424113 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:15:23.019599 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Create backup started, backup: c4667eed-7f57-4744-985f-03f4b9dec074 volume: 3a4a0854-8b3f-4537-abf3-52b0cc5e779a.
Apr 09 21:15:23.035821 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:15:22Z,data_timestamp=2022-04-09T21:15:22Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=c4667eed-7f57-4744-985f-03f4b9dec074,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='a2e537a0863d43df811118a31e20609c',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='8303163dfe6c44dc8552919975c72eba',volume_id=3a4a0854-8b3f-4537-abf3-52b0cc5e779a)
Apr 09 21:15:23.106431 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:15:23.111391 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.143720 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.032s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:23.144269 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[53d21a6a-906b-4eb1-b862-3fa3d4bda8f5]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:23.148793 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.166935 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[900a21f7-51c9-418b-b338-b2d7d829ba19]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:15:23.167546 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:15:23.169141 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[900a21f7-51c9-418b-b338-b2d7d829ba19]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:23.172082 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.193634 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[8d1c834a-afb1-41e3-bd43-b93b0159581d]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:15:23.194231 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:15:23.195968 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[8d1c834a-afb1-41e3-bd43-b93b0159581d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:23.197833 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.234045 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.036s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:23.234617 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[99754817-ff58-4518-a96f-395cf0a4a64b]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:23.236279 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.268604 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.032s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:23.269270 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[f55036ff-96fe-4f8d-bf31-efa6ee0d7f48]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:23.272098 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.309425 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.038s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:23.309716 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[c34580ab-0e16-42f3-893e-0950b7366c70]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:23.310692 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:23.367321 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] CMD "nvme version" returned: 0 in 0.056s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:23.370546 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:15:23.371052 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:15:23.371526 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:15:23.372303 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] <== get_connector_properties: return (262ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:15:23.664235 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-3f6514b9-f05a-423b-9d37-299800de2387 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:15:25.215978 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:15:25.217508 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee203700>, 'connection_properties': {'name': 'volumes/volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3a4a0854-8b3f-4537-abf3-52b0cc5e779a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:15:25.218565 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:15:25.370538 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] <== connect_volume: return (152ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed83e4f0>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:15:25.405710 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Starting backup of volume='3a4a0854-8b3f-4537-abf3-52b0cc5e779a'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:15:25.407446 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Volume file is RBD: attempting optimized backup {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}}
Apr 09 21:15:25.496486 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Creating base image 'volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' {{(pid=134916) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}}
Apr 09 21:15:25.497254 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:15:25.591443 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Using --from-snap 'None' for incremental backup of volume 3a4a0854-8b3f-4537-abf3-52b0cc5e779a. {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}}
Apr 09 21:15:25.592146 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Creating backup snapshot='backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657' {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}}
Apr 09 21:15:26.724404 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Performing differential transfer from 'volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a' to 'volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:15:26.725108 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_dm5iru3p --pool volumes volumes/volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a@backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:15:26.725206 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:15:35.924456 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Differential backup transfer completed in 9.2001s {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}}
Apr 09 21:15:35.925288 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:15:35.958520 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:15:35.960915 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:15:35.977058 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:15:35.978835 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:15:36.006613 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}}
Apr 09 21:15:36.009408 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Backing up metadata for volume 3a4a0854-8b3f-4537-abf3-52b0cc5e779a. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:15:36.159554 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Backup 'c4667eed-7f57-4744-985f-03f4b9dec074' of volume 3a4a0854-8b3f-4537-abf3-52b0cc5e779a finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:15:36.172838 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee203700>, 'connection_properties': {'name': 'volumes/volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3a4a0854-8b3f-4537-abf3-52b0cc5e779a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed83e4f0>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:15:36.233958 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] <== disconnect_volume: return (58ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:15:36.331404 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-6be3d951-d3c0-42d8-8a72-15c745b943c5 tempest-TestVolumeBackupRestore-1877640871 None] Create backup finished. backup: c4667eed-7f57-4744-985f-03f4b9dec074.
Apr 09 21:15:38.427369 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:15:38.427369 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:15:38.465781 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Restore backup started, backup: c4667eed-7f57-4744-985f-03f4b9dec074 volume: 77d8a986-eb34-4d3a-9ad7-5ad1b19954dd.
Apr 09 21:15:38.491610 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:15:38.497418 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.507667 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.014s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:38.508000 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[fc6ed4c4-7129-4375-b6e5-17f00921426e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:38.509423 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.527517 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[cc3cf3ad-2f19-4661-8867-e58caddd46d1]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:15:38.528127 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:15:38.529909 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[cc3cf3ad-2f19-4661-8867-e58caddd46d1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:38.531263 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.564989 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0c9993ec-d34f-484c-8b1b-b4b8b31f9c45]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:15:38.565605 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:15:38.567229 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[0c9993ec-d34f-484c-8b1b-b4b8b31f9c45]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:38.568809 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.606893 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.037s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:38.607447 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[f8d2e1aa-b96e-4ece-834f-e7ee3ce59a26]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:38.611684 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.653084 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.043s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:38.653906 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[6f0552c2-a327-430f-b440-dff9a79c7b97]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:38.656096 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.686738 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.029s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:38.687224 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[34e764b7-0887-436e-8e7f-7c566e3387bd]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:15:38.688266 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:15:38.771709 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] CMD "nvme version" returned: 0 in 0.080s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:15:38.773576 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:15:38.775166 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:15:38.775839 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:15:38.776786 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] <== get_connector_properties: return (284ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:15:40.711917 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:15:40.718335 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee192850>, 'connection_properties': {'name': 'volumes/volume-77d8a986-eb34-4d3a-9ad7-5ad1b19954dd', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '77d8a986-eb34-4d3a-9ad7-5ad1b19954dd', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:15:40.720040 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:15:40.813089 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] <== connect_volume: return (93ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed8942e0>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:15:40.823336 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:15:40.859910 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Starting restore from Ceph backup=c4667eed-7f57-4744-985f-03f4b9dec074 to volume=volume-77d8a986-eb34-4d3a-9ad7-5ad1b19954dd {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}}
Apr 09 21:15:41.018989 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Looking for snapshot of backup base 'volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:15:41.019388 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Found snapshot 'backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:15:41.031483 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Volume file is RBD. {{(pid=134916) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}}
Apr 09 21:15:41.081409 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Attempting differential restore. {{(pid=134916) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}}
Apr 09 21:15:41.081963 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Attempting incremental restore from base='volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' snap='backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657' {{(pid=134916) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}}
Apr 09 21:15:41.082279 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Performing differential transfer from 'volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' to 'volume-77d8a986-eb34-4d3a-9ad7-5ad1b19954dd' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:15:41.082679 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074@backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:15:41.083051 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_l4v811ya --pool volumes - volumes/volume-77d8a986-eb34-4d3a-9ad7-5ad1b19954dd' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:15:49.075526 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Restore transfer completed in 7.9927s {{(pid=134916) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}}
Apr 09 21:15:49.076973 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}}
Apr 09 21:15:49.131232 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Restoring volume base metadata {{(pid=134916) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Apr 09 21:15:49.164645 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] No metadata of type 'volume-metadata' to restore {{(pid=134916) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Apr 09 21:15:49.165417 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Restoring volume glance metadata {{(pid=134916) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}}
Apr 09 21:15:49.444735 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Restore to volume 77d8a986-eb34-4d3a-9ad7-5ad1b19954dd finished successfully. {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}}
Apr 09 21:15:49.447751 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee192850>, 'connection_properties': {'name': 'volumes/volume-77d8a986-eb34-4d3a-9ad7-5ad1b19954dd', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '77d8a986-eb34-4d3a-9ad7-5ad1b19954dd', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed8942e0>}, 'force': True, ': False}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:15:49.491628 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] <== disconnect_volume: return (41ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:15:49.583152 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.manager [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Encryption key ID for volume 77d8a986-eb34-4d3a-9ad7-5ad1b19954dd already matches encryption key ID in backup c4667eed-7f57-4744-985f-03f4b9dec074. {{(pid=134916) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}}
Apr 09 21:15:49.673407 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-757492c6-7efc-4c22-8582-5fa1b5125fa0 tempest-TestVolumeBackupRestore-1877640871 None] Finished restoring backup c4667eed-7f57-4744-985f-03f4b9dec074 to volume 77d8a986-eb34-4d3a-9ad7-5ad1b19954dd.
Apr 09 21:16:38.422417 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:16:38.423673 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:17:38.420437 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:17:38.421210 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:17:42.106909 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Delete backup started, backup: c4667eed-7f57-4744-985f-03f4b9dec074.
Apr 09 21:17:42.108991 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Delete started for backup=c4667eed-7f57-4744-985f-03f4b9dec074 {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:17:42.109433 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Trying diff format basename='volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' for backup base image of volume 3a4a0854-8b3f-4537-abf3-52b0cc5e779a. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:17:42.225729 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Looking for snapshot of backup base 'volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:17:42.226130 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Found snapshot 'backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:17:42.226588 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Deleting backup snapshot='backup.c4667eed-7f57-4744-985f-03f4b9dec074.snap.1649538925.5914657' {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}}
Apr 09 21:17:43.159997 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Deleting backup base image='volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' of volume 3a4a0854-8b3f-4537-abf3-52b0cc5e779a.
Apr 09 21:17:43.683787 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-3f6514b9-f05a-423b-9d37-299800de2387 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:17:44.540366 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Base backup image='volume-3a4a0854-8b3f-4537-abf3-52b0cc5e779a.backup.c4667eed-7f57-4744-985f-03f4b9dec074' of volume 3a4a0854-8b3f-4537-abf3-52b0cc5e779a deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:17:44.621534 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Delete of backup 'c4667eed-7f57-4744-985f-03f4b9dec074' for volume '3a4a0854-8b3f-4537-abf3-52b0cc5e779a' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:17:44.670519 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Created reservations ['8f99cc0d-bbe2-4ceb-bab2-64b6b47be792', '75136aad-2ec9-443a-8f15-9921831a1347'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:17:44.718832 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-1e1d0677-c6ca-42f1-945d-c872728ebad5 tempest-TestVolumeBackupRestore-1877640871 None] Delete backup finished, backup c4667eed-7f57-4744-985f-03f4b9dec074 deleted.
Apr 09 21:18:38.417271 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:18:38.418565 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:19:38.419411 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:19:38.420114 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:20:38.416633 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:20:38.417286 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:21:38.416723 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:21:38.417378 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:22:15.885767 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Create backup started, backup: 857bb7fb-7e8d-4c05-ae48-7588cc36804c volume: 7fe19ff5-f30f-493a-bfb0-c7288094f340.
Apr 09 21:22:15.912653 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:22:15Z,data_timestamp=2022-04-09T21:22:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-401991413',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=857bb7fb-7e8d-4c05-ae48-7588cc36804c,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='b3cd8a38803f4a8dbceba2c5a64f997f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='dead4209e2844000bf7d563164daa14a',volume_id=7fe19ff5-f30f-493a-bfb0-c7288094f340)
Apr 09 21:22:15.961331 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:22:15.964905 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:15.975551 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.012s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:22:15.975959 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[0106310b-1ffa-44ff-ba4e-3677a4956f35]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:22:15.977287 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:15.986566 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[986a18bf-1534-4d0d-b120-8f6db274a02d]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:22:15.987499 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:22:15.988790 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[986a18bf-1534-4d0d-b120-8f6db274a02d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:22:15.991527 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:16.001419 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ffdc8491-7002-4096-820a-35745ec278d5]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:22:16.002808 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:22:16.031762 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[ffdc8491-7002-4096-820a-35745ec278d5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:22:16.040698 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:16.061222 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.025s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:22:16.062373 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[9a65940c-b678-4021-9860-e12a5fbbecc0]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:22:16.067116 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:16.098698 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.034s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:22:16.099281 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[b7dd09ce-198a-412b-99b2-e4267b23fcbc]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:22:16.100854 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:16.129834 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.028s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:22:16.130606 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[7d055a7a-5e48-454c-973a-8f4a6a5e2a8a]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:22:16.131746 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:22:16.192239 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] CMD "nvme version" returned: 0 in 0.060s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:22:16.194418 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:22:16.194536 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:22:16.195034 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:22:16.195678 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] <== get_connector_properties: return (233ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:22:17.091039 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:22:17.092700 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed83e640>, 'connection_properties': {'name': 'volumes/volume-7fe19ff5-f30f-493a-bfb0-c7288094f340', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7fe19ff5-f30f-493a-bfb0-c7288094f340', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:22:17.103895 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:22:17.240977 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] <== connect_volume: return (147ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ee246160>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:22:17.251683 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:22:17.310052 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Starting backup of volume='7fe19ff5-f30f-493a-bfb0-c7288094f340'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:22:17.310946 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Volume file is RBD: attempting optimized backup {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}}
Apr 09 21:22:17.386165 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Creating base image 'volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' {{(pid=134916) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}}
Apr 09 21:22:17.387493 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:22:17.465387 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Using --from-snap 'None' for incremental backup of volume 7fe19ff5-f30f-493a-bfb0-c7288094f340. {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}}
Apr 09 21:22:17.465908 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Creating backup snapshot='backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c.snap.1649539337.4654093' {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}}
Apr 09 21:22:18.446932 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Performing differential transfer from 'volume-7fe19ff5-f30f-493a-bfb0-c7288094f340' to 'volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:22:18.448029 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_zutpv3kb --pool volumes volumes/volume-7fe19ff5-f30f-493a-bfb0-c7288094f340@backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c.snap.1649539337.4654093 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:22:18.448755 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:22:19.503176 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Differential backup transfer completed in 1.0555s {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}}
Apr 09 21:22:19.504148 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:22:19.521997 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:22:19.522669 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:22:19.531810 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:22:19.532564 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:22:19.540049 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] No metadata type 'volume-glance-metadata' available {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Apr 09 21:22:19.540952 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Backing up metadata for volume 7fe19ff5-f30f-493a-bfb0-c7288094f340. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:22:19.598559 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Backup '857bb7fb-7e8d-4c05-ae48-7588cc36804c' of volume 7fe19ff5-f30f-493a-bfb0-c7288094f340 finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:22:19.600446 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed83e640>, 'connection_properties': {'name': 'volumes/volume-7fe19ff5-f30f-493a-bfb0-c7288094f340', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7fe19ff5-f30f-493a-bfb0-c7288094f340', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ee246160>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:22:19.641961 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] <== disconnect_volume: return (40ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:22:19.765996 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-31adc7ba-13b7-4c16-a1f3-45de6418986a tempest-VolumesBackupsV39Test-222054102 None] Create backup finished. backup: 857bb7fb-7e8d-4c05-ae48-7588cc36804c.
Apr 09 21:22:20.264474 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Delete backup started, backup: 857bb7fb-7e8d-4c05-ae48-7588cc36804c.
Apr 09 21:22:20.266563 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Delete started for backup=857bb7fb-7e8d-4c05-ae48-7588cc36804c {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:22:20.267149 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Trying diff format basename='volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' for backup base image of volume 7fe19ff5-f30f-493a-bfb0-c7288094f340. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:22:20.396338 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Looking for snapshot of backup base 'volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:22:20.398383 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Found snapshot 'backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c.snap.1649539337.4654093' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:22:20.398994 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Deleting backup snapshot='backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c.snap.1649539337.4654093' {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}}
Apr 09 21:22:21.523693 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Deleting backup base image='volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' of volume 7fe19ff5-f30f-493a-bfb0-c7288094f340.
Apr 09 21:22:21.790573 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Base backup image='volume-7fe19ff5-f30f-493a-bfb0-c7288094f340.backup.857bb7fb-7e8d-4c05-ae48-7588cc36804c' of volume 7fe19ff5-f30f-493a-bfb0-c7288094f340 deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:22:21.858846 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Delete of backup '857bb7fb-7e8d-4c05-ae48-7588cc36804c' for volume '7fe19ff5-f30f-493a-bfb0-c7288094f340' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:22:21.864785 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:22:21.895993 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Created reservations ['70d63acb-2713-43f1-be24-fd36c0439bb7', '3e99a7e5-b77f-414b-a226-7c20a23fa9c2'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:22:21.930243 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-795f6d1d-907c-4a30-81c1-e96128f4089f tempest-VolumesBackupsV39Test-222054102 None] Delete backup finished, backup 857bb7fb-7e8d-4c05-ae48-7588cc36804c deleted.
Apr 09 21:23:38.415934 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:23:38.416616 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:23:44.571916 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Create backup started, backup: ac0a6725-6e9b-4166-94fc-8c69738477b5 volume: ad4054d0-75a9-4219-b2d3-841907154627.
Apr 09 21:23:44.581904 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:23:44Z,data_timestamp=2022-04-09T21:23:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1326047191',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=ac0a6725-6e9b-4166-94fc-8c69738477b5,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='7346ba2c9aef4d22ae03e0fa2bd3e7e1',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f417c3f302d54800921c418c34e3d694',volume_id=ad4054d0-75a9-4219-b2d3-841907154627)
Apr 09 21:23:44.620126 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:44.621781 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.630740 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:44.631093 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[4d900721-e5dd-430c-991e-069cb578c28e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:44.632431 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.639861 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e4559e7d-54b4-4226-892e-f44ec0d33058]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:23:44.640480 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:23:44.641555 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[e4559e7d-54b4-4226-892e-f44ec0d33058]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:44.642145 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.647992 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[131a1dd4-68c9-4d19-83ae-95d2dde26909]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:23:44.648611 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:23:44.649653 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[131a1dd4-68c9-4d19-83ae-95d2dde26909]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:44.650458 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.673429 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.023s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:44.673814 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[2a889676-f2f0-428c-9420-c969d902e8cc]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:44.675501 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.684534 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:44.684968 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[885a508b-8fa2-4665-b0bc-36f29693af07]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:44.686471 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.693374 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:44.693704 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[9cb2b2ee-26fa-4227-afe6-1bfa441cc22b]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:44.694495 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:44.712522 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:44.714318 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:23:44.714725 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:23:44.715103 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:23:44.715753 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] <== get_connector_properties: return (94ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:45.430202 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:23:45.431717 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee0f8820>, 'connection_properties': {'name': 'volumes/volume-ad4054d0-75a9-4219-b2d3-841907154627', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ad4054d0-75a9-4219-b2d3-841907154627', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:45.432798 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:23:45.481332 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] <== connect_volume: return (49ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed884730>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:45.489465 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:23:45.510326 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Starting backup of volume='ad4054d0-75a9-4219-b2d3-841907154627'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:23:45.511339 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Volume file is RBD: attempting optimized backup {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}}
Apr 09 21:23:45.531173 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Creating base image 'volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' {{(pid=134916) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}}
Apr 09 21:23:45.531679 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:23:45.577455 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Using --from-snap 'None' for incremental backup of volume ad4054d0-75a9-4219-b2d3-841907154627. {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}}
Apr 09 21:23:45.578017 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Creating backup snapshot='backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844' {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}}
Apr 09 21:23:45.878669 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Performing differential transfer from 'volume-ad4054d0-75a9-4219-b2d3-841907154627' to 'volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:23:45.879098 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_qjbetzzo --pool volumes volumes/volume-ad4054d0-75a9-4219-b2d3-841907154627@backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:23:45.879473 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:23:46.915448 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Differential backup transfer completed in 1.0364s {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}}
Apr 09 21:23:46.916163 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:23:46.930086 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:23:46.930539 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:23:46.937811 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:23:46.938200 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:23:46.945105 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] No metadata type 'volume-glance-metadata' available {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Apr 09 21:23:46.945787 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Backing up metadata for volume ad4054d0-75a9-4219-b2d3-841907154627. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:23:46.988819 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Backup 'ac0a6725-6e9b-4166-94fc-8c69738477b5' of volume ad4054d0-75a9-4219-b2d3-841907154627 finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:23:46.990757 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee0f8820>, 'connection_properties': {'name': 'volumes/volume-ad4054d0-75a9-4219-b2d3-841907154627', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ad4054d0-75a9-4219-b2d3-841907154627', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed884730>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:47.012704 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] <== disconnect_volume: return (21ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:47.070293 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d88ec2f7-a473-49b4-a5ca-144f169a6e68 tempest-VolumesBackupsAdminTest-673043516 None] Create backup finished. backup: ac0a6725-6e9b-4166-94fc-8c69738477b5.
Apr 09 21:23:47.808336 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-07589a12-9cb5-4851-86f2-5b6cfcc4eafd tempest-VolumesBackupsAdminTest-1610687721 None] Export record started, backup: ac0a6725-6e9b-4166-94fc-8c69738477b5.
Apr 09 21:23:47.810392 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-07589a12-9cb5-4851-86f2-5b6cfcc4eafd tempest-VolumesBackupsAdminTest-1610687721 None] Export record finished, backup ac0a6725-6e9b-4166-94fc-8c69738477b5 exported.
Apr 09 21:23:47.944791 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-2c138142-f1d3-4cd6-936c-cd8e7cde5dae tempest-VolumesBackupsAdminTest-1610687721 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiOGZiOWYwODAtYmRiMC00YzZiLTgxMDYtZDAxMTliOGJmOWRhIiwgInVzZXJfaWQiOiAiZjQxN2MzZjMwMmQ1NDgwMDkyMWM0MThjMzRlM2Q2OTQiLCAicHJvamVjdF9pZCI6ICI3MzQ2YmEyYzlhZWY0ZDIyYWUwM2UwZmEyYmQzZTdlMSIsICJ2b2x1bWVfaWQiOiAiYWQ0MDU0ZDAtNzVhOS00MjE5LWIyZDMtODQxOTA3MTU0NjI3IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xMzI2MDQ3MTkxIiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLWFkNDA1NGQwLTc1YTktNDIxOS1iMmQzLTg0MTkwNzE1NDYyNy5iYWNrdXAuYWMwYTY3MjUtNmU5Yi00MTY2LTk0ZmMtOGM2OTczODQ3N2I1XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wNC0wOVQyMToyMzo0NFoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wNC0wOVQyMToyMzo0NFoiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTA0LTA5VDIxOjIzOjQ3WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=.
Apr 09 21:23:47.955577 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-2c138142-f1d3-4cd6-936c-cd8e7cde5dae tempest-VolumesBackupsAdminTest-1610687721 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:23:47.995758 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-2c138142-f1d3-4cd6-936c-cd8e7cde5dae tempest-VolumesBackupsAdminTest-1610687721 None] Import record id 8fb9f080-bdb0-4c6b-8106-d0119b8bf9da metadata from driver finished.
Apr 09 21:23:50.380339 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Restore backup started, backup: ac0a6725-6e9b-4166-94fc-8c69738477b5 volume: a6e6c23b-1f26-4f8e-8459-4296e8b8847f.
Apr 09 21:23:50.394212 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:50.395748 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.404193 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:50.404537 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[69861ea3-fb08-456c-a56b-4a14946617f5]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:50.405929 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.411942 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[bf2fae9b-c9ec-4ab8-b7b0-74f454542a63]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:23:50.412530 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:23:50.413564 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[bf2fae9b-c9ec-4ab8-b7b0-74f454542a63]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:50.414221 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.420114 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[dee2b1ab-bf71-41fd-b2c0-0e50693a58e4]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:23:50.420665 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:23:50.421786 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[dee2b1ab-bf71-41fd-b2c0-0e50693a58e4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:50.424005 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.432474 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:50.432832 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[a0b65de4-70c9-415a-9aa9-aa98ac73e54a]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:50.434243 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.445375 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.011s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:50.445714 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[a297611a-71ef-44e1-a321-0362aa9c08ae]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:50.446926 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.455077 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:50.455637 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[836cd7ed-ed00-4122-89de-16b7d3417233]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:50.456458 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:50.474657 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:50.476449 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:23:50.476851 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:23:50.477282 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:23:50.477925 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] <== get_connector_properties: return (82ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:51.143814 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:23:51.145333 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee454160>, 'connection_properties': {'name': 'volumes/volume-a6e6c23b-1f26-4f8e-8459-4296e8b8847f', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a6e6c23b-1f26-4f8e-8459-4296e8b8847f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:51.146438 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:23:51.189308 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] <== connect_volume: return (43ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed7fae20>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:51.196659 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:23:51.209687 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Starting restore from Ceph backup=ac0a6725-6e9b-4166-94fc-8c69738477b5 to volume=volume-a6e6c23b-1f26-4f8e-8459-4296e8b8847f {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}}
Apr 09 21:23:51.271247 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Looking for snapshot of backup base 'volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:23:51.271637 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Found snapshot 'backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:23:51.280060 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Volume file is RBD. {{(pid=134916) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}}
Apr 09 21:23:51.323061 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Attempting differential restore. {{(pid=134916) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}}
Apr 09 21:23:51.323590 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Attempting incremental restore from base='volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' snap='backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844' {{(pid=134916) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}}
Apr 09 21:23:51.323997 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Performing differential transfer from 'volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' to 'volume-a6e6c23b-1f26-4f8e-8459-4296e8b8847f' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:23:51.324406 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5@backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:23:51.324785 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_w3nmry7r --pool volumes - volumes/volume-a6e6c23b-1f26-4f8e-8459-4296e8b8847f' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:23:52.033555 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Restore transfer completed in 0.7087s {{(pid=134916) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}}
Apr 09 21:23:52.034547 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}}
Apr 09 21:23:52.062447 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Restoring volume base metadata {{(pid=134916) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Apr 09 21:23:52.086833 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] No metadata of type 'volume-metadata' to restore {{(pid=134916) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Apr 09 21:23:52.087577 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=134916) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Apr 09 21:23:52.091236 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Restore to volume a6e6c23b-1f26-4f8e-8459-4296e8b8847f finished successfully. {{(pid=134916) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}}
Apr 09 21:23:52.093401 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ee454160>, 'connection_properties': {'name': 'volumes/volume-a6e6c23b-1f26-4f8e-8459-4296e8b8847f', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a6e6c23b-1f26-4f8e-8459-4296e8b8847f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed7fae20>}, 'force': True, ': False}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:52.106377 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] <== disconnect_volume: return (12ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:52.153500 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.manager [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Encryption key ID for volume a6e6c23b-1f26-4f8e-8459-4296e8b8847f already matches encryption key ID in backup ac0a6725-6e9b-4166-94fc-8c69738477b5. {{(pid=134916) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}}
Apr 09 21:23:52.187626 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-a8ff5f5c-59cc-4611-8109-82a24ce2dc1f tempest-VolumesBackupsAdminTest-673043516 None] Finished restoring backup ac0a6725-6e9b-4166-94fc-8c69738477b5 to volume a6e6c23b-1f26-4f8e-8459-4296e8b8847f.
Apr 09 21:23:52.724924 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Delete backup started, backup: 8fb9f080-bdb0-4c6b-8106-d0119b8bf9da.
Apr 09 21:23:52.726981 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Delete started for backup=8fb9f080-bdb0-4c6b-8106-d0119b8bf9da {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:23:52.727412 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Trying diff format basename='volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' for backup base image of volume ad4054d0-75a9-4219-b2d3-841907154627. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:23:52.825214 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Looking for snapshot of backup base 'volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:23:52.827215 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Backup '8fb9f080-bdb0-4c6b-8106-d0119b8bf9da' has no snapshot {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:962}}
Apr 09 21:23:52.827838 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] No backup snapshot to delete {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}}
Apr 09 21:23:52.850957 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Backup base image of volume ad4054d0-75a9-4219-b2d3-841907154627 still has 1 snapshots so skipping base image delete.
Apr 09 21:23:52.933106 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Metadata backup object 'backup.8fb9f080-bdb0-4c6b-8106-d0119b8bf9da.meta' not found - ignoring {{(pid=134916) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:162}}
Apr 09 21:23:52.938932 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Delete of backup '8fb9f080-bdb0-4c6b-8106-d0119b8bf9da' for volume 'ad4054d0-75a9-4219-b2d3-841907154627' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:23:52.946283 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:23:52.979237 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Created reservations ['833652ba-27c2-48d2-bc9c-a3c43cdb9bd8', '6abcaec3-6dac-41d7-9089-25f868bd6cfc'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:23:53.032092 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-4dd03d14-5b4b-47c0-8d4a-3c18db9b026d tempest-VolumesBackupsAdminTest-1610687721 None] Delete backup finished, backup 8fb9f080-bdb0-4c6b-8106-d0119b8bf9da deleted.
Apr 09 21:23:53.817149 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Delete backup started, backup: ac0a6725-6e9b-4166-94fc-8c69738477b5.
Apr 09 21:23:53.818501 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Delete started for backup=ac0a6725-6e9b-4166-94fc-8c69738477b5 {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:23:53.819188 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Trying diff format basename='volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' for backup base image of volume ad4054d0-75a9-4219-b2d3-841907154627. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:23:53.874475 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Looking for snapshot of backup base 'volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:23:53.874890 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Found snapshot 'backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:23:53.875274 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Deleting backup snapshot='backup.ac0a6725-6e9b-4166-94fc-8c69738477b5.snap.1649539425.5774844' {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}}
Apr 09 21:23:54.045447 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Deleting backup base image='volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' of volume ad4054d0-75a9-4219-b2d3-841907154627.
Apr 09 21:23:54.290405 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Base backup image='volume-ad4054d0-75a9-4219-b2d3-841907154627.backup.ac0a6725-6e9b-4166-94fc-8c69738477b5' of volume ad4054d0-75a9-4219-b2d3-841907154627 deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:23:54.327687 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Delete of backup 'ac0a6725-6e9b-4166-94fc-8c69738477b5' for volume 'ad4054d0-75a9-4219-b2d3-841907154627' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:23:54.333426 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:23:54.359239 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Created reservations ['d6ce0396-3d8d-4f96-ac51-76455d6d3111', '8310e34c-7429-4bb7-945b-e26198aabd7b'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:23:54.390085 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d0fd63ee-5905-4428-bb9e-feea43080f8e tempest-VolumesBackupsAdminTest-673043516 None] Delete backup finished, backup ac0a6725-6e9b-4166-94fc-8c69738477b5 deleted.
Apr 09 21:23:56.350772 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Create backup started, backup: 362e602c-e158-4b11-8c15-4ae47070396e volume: 476106a5-8968-4546-bada-3beb55b5e27a.
Apr 09 21:23:56.359330 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:23:56Z,data_timestamp=2022-04-09T21:23:56Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-611943603',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=362e602c-e158-4b11-8c15-4ae47070396e,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='7346ba2c9aef4d22ae03e0fa2bd3e7e1',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f417c3f302d54800921c418c34e3d694',volume_id=476106a5-8968-4546-bada-3beb55b5e27a)
Apr 09 21:23:56.399465 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:56.400957 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.410437 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:56.410790 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[fec536f9-a967-47bc-89b7-1a02dca39b2f]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:56.414317 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.419362 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[61d09642-8ea4-4cab-af7b-06ab7a0c70ee]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:23:56.419816 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:23:56.420698 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[61d09642-8ea4-4cab-af7b-06ab7a0c70ee]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:56.421987 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.428246 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[cc980dfb-e80d-436b-9208-2fa793856c0a]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:23:56.429336 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:23:56.430217 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[cc980dfb-e80d-436b-9208-2fa793856c0a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:56.431391 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.440847 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:56.441297 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[b51bf88e-e1eb-4648-9abe-96269c519766]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:56.444580 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.465312 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.020s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:56.465643 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[172f7c87-8b55-4ed4-b88d-b4b2f7311ab4]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:56.467321 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.476458 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:56.477161 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[5ef5e9ce-2780-4141-b263-8cfce32de787]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:23:56.478152 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:23:56.497871 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:23:56.499829 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:23:56.500274 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:23:56.500718 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:23:56.501495 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] <== get_connector_properties: return (101ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:57.198726 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:23:57.200243 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed7fa8b0>, 'connection_properties': {'name': 'volumes/volume-476106a5-8968-4546-bada-3beb55b5e27a', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '476106a5-8968-4546-bada-3beb55b5e27a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:57.201346 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:23:57.245892 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] <== connect_volume: return (44ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed7fad90>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:57.253321 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:23:57.275009 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Starting backup of volume='476106a5-8968-4546-bada-3beb55b5e27a'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:23:57.275991 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Volume file is RBD: attempting optimized backup {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}}
Apr 09 21:23:57.296604 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Creating base image 'volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' {{(pid=134916) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}}
Apr 09 21:23:57.297018 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:23:57.341261 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Using --from-snap 'None' for incremental backup of volume 476106a5-8968-4546-bada-3beb55b5e27a. {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}}
Apr 09 21:23:57.341682 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Creating backup snapshot='backup.362e602c-e158-4b11-8c15-4ae47070396e.snap.1649539437.3412707' {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}}
Apr 09 21:23:57.583085 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Performing differential transfer from 'volume-476106a5-8968-4546-bada-3beb55b5e27a' to 'volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' {{(pid=134916) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}}
Apr 09 21:23:57.583749 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_ctem2wx9 --pool volumes volumes/volume-476106a5-8968-4546-bada-3beb55b5e27a@backup.362e602c-e158-4b11-8c15-4ae47070396e.snap.1649539437.3412707 -' into... {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}}
Apr 09 21:23:57.583932 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' {{(pid=134916) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}}
Apr 09 21:23:58.639024 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Differential backup transfer completed in 1.0554s {{(pid=134916) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}}
Apr 09 21:23:58.639619 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:23:58.655141 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:23:58.655559 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:23:58.663342 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:23:58.663975 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:23:58.683682 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] No metadata type 'volume-glance-metadata' available {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Apr 09 21:23:58.684890 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Backing up metadata for volume 476106a5-8968-4546-bada-3beb55b5e27a. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:23:58.758107 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Backup '362e602c-e158-4b11-8c15-4ae47070396e' of volume 476106a5-8968-4546-bada-3beb55b5e27a finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:23:58.760424 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed7fa8b0>, 'connection_properties': {'name': 'volumes/volume-476106a5-8968-4546-bada-3beb55b5e27a', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '476106a5-8968-4546-bada-3beb55b5e27a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed7fad90>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:23:58.785524 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] <== disconnect_volume: return (24ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:23:58.841475 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-105f73d4-aca4-41d5-b3d8-5c6ebacfa49e tempest-VolumesBackupsAdminTest-673043516 None] Create backup finished. backup: 362e602c-e158-4b11-8c15-4ae47070396e.
Apr 09 21:23:59.516930 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d3195068-93be-4d89-90eb-2e8512b61582 tempest-VolumesBackupsAdminTest-1610687721 None] Reset backup status started, backup_id: 362e602c-e158-4b11-8c15-4ae47070396e, status: error.
Apr 09 21:23:59.517519 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-d3195068-93be-4d89-90eb-2e8512b61582 tempest-VolumesBackupsAdminTest-1610687721 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver.
Apr 09 21:23:59.593103 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Delete backup started, backup: 362e602c-e158-4b11-8c15-4ae47070396e.
Apr 09 21:23:59.594586 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Delete started for backup=362e602c-e158-4b11-8c15-4ae47070396e {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:23:59.595048 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Trying diff format basename='volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' for backup base image of volume 476106a5-8968-4546-bada-3beb55b5e27a. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:23:59.643305 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Looking for snapshot of backup base 'volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:23:59.643931 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Found snapshot 'backup.362e602c-e158-4b11-8c15-4ae47070396e.snap.1649539437.3412707' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}}
Apr 09 21:23:59.644020 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Deleting backup snapshot='backup.362e602c-e158-4b11-8c15-4ae47070396e.snap.1649539437.3412707' {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}}
Apr 09 21:24:00.667604 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Deleting backup base image='volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' of volume 476106a5-8968-4546-bada-3beb55b5e27a.
Apr 09 21:24:00.881738 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Base backup image='volume-476106a5-8968-4546-bada-3beb55b5e27a.backup.362e602c-e158-4b11-8c15-4ae47070396e' of volume 476106a5-8968-4546-bada-3beb55b5e27a deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:24:00.924147 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Delete of backup '362e602c-e158-4b11-8c15-4ae47070396e' for volume '476106a5-8968-4546-bada-3beb55b5e27a' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:24:00.929706 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:24:00.953382 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Created reservations ['b330096d-6aba-437b-82e3-e0bfca73ac7c', '48ef1b5d-ae18-4016-ad74-4d09232ad6b7'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:24:00.982647 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-5919c4cf-7f1c-49d1-9687-e3e9cb41ffa1 tempest-VolumesBackupsAdminTest-673043516 None] Delete backup finished, backup 362e602c-e158-4b11-8c15-4ae47070396e deleted.
Apr 09 21:24:38.422386 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:24:38.423405 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:25:38.421106 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:25:38.422180 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:26:38.419932 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:26:38.420433 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:27:38.420262 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:27:38.420939 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:28:38.417484 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:28:38.418719 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:29:38.415617 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:29:38.416296 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:30:35.504651 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Create backup started, backup: f41cb63e-911f-4569-881d-c775f1c4df59 volume: fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b snapshot: 9cdf63cf-ccc4-49b8-a938-a3945341defb.
Apr 09 21:30:35.512473 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-09T21:30:35Z,data_timestamp=2022-04-09T21:30:32Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-1277760299',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=f41cb63e-911f-4569-881d-c775f1c4df59,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='3e1e5cc7222142ab82cd6a857f7be3de',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='9cdf63cf-ccc4-49b8-a938-a3945341defb',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f858bd78e1794185a908d3372a7359dd',volume_id=fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b)
Apr 09 21:30:35.786538 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:30:35.787873 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.796243 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:30:35.796578 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[541f139a-f647-4315-8a5f-056502b98f72]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a8a3214ddfa3\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:30:35.798275 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.805408 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7c37131c-a77b-499d-a593-e16096a5e26a]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:30:35.806349 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:30:35.807348 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c37131c-a77b-499d-a593-e16096a5e26a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:30:35.807796 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.814926 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ac3bc341-0cb6-4c9e-998c-609d7a628241]: [Errno 2] No such file or directory: 'systool' {{(pid=159908) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}}
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]: Traceback (most recent call last):
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     ret = func(*f_args, **f_kwargs)
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     return func(*args, **kwargs)
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     return putils.execute(on_execute=on_execute,
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     obj = subprocess.Popen(cmd,
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     self._execute_child(args, executable, preexec_fn, close_fds,
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]:     raise child_exception_type(errno_num, err_msg, err_filename)
Apr 09 21:30:35.815477 ncdv-master-1 cinder-backup[134916]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Apr 09 21:30:35.816489 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[ac3bc341-0cb6-4c9e-998c-609d7a628241]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:30:35.817301 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.826644 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:30:35.826999 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[fc4475a0-0670-4048-832c-e7582b6e35cf]: (4, ('/dev/sda1\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:30:35.829186 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.839664 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:30:35.840006 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[aed4f836-08ad-4694-8c63-e92ec8ceef39]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:30:35.841267 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.849426 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=159908) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:30:35.849715 ncdv-master-1 cinder-backup[134916]: DEBUG oslo.privsep.daemon [-] privsep: reply[8f6f9e93-aa70-4457-8002-6e7f7f73de24]: (4, ('34112a9b-6ce0-1846-9799-b8160470743a\n', '')) {{(pid=159908) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}
Apr 09 21:30:35.850456 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Running cmd (subprocess): nvme version {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Apr 09 21:30:35.871347 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_concurrency.processutils [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] CMD "nvme version" returned: 0 in 0.021s {{(pid=134916) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Apr 09 21:30:35.873135 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134916) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}}
Apr 09 21:30:35.873512 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}}
Apr 09 21:30:35.873879 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.lightos [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f dsc:  {{(pid=134916) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}}
Apr 09 21:30:35.874697 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.utils [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] <== get_connector_properties: return (87ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a8a3214ddfa3', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '34112a9b-6ce0-1846-9799-b8160470743a', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:c2dae032-cd4e-4f32-a8ec-6ebc34934b2f', 'found_dsc': ''} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:30:36.556086 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connector [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Factory for rbd on None {{(pid=134916) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}}
Apr 09 21:30:36.557678 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed893370>, 'connection_properties': {'name': 'volumes/volume-a5c65ac7-7c90-409b-a38a-0d7304ee17bf', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a5c65ac7-7c90-409b-a38a-0d7304ee17bf', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:30:36.558803 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.linuxrbd [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] opening connection to ceph cluster (timeout=-1). {{(pid=134916) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Apr 09 21:30:36.618156 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] <== connect_volume: return (59ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed888b20>} {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:30:36.626510 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:30:36.647470 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Starting backup of volume='fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b'. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}}
Apr 09 21:30:36.672623 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134916) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}}
Apr 09 21:30:36.673116 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Creating backup base image='volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.base' for volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b. {{(pid=134916) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:875}}
Apr 09 21:30:36.706191 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Copying data from volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b. {{(pid=134916) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:887}}
Apr 09 21:30:36.725635 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferring data between 'volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b' and 'volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.base' {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:417}}
Apr 09 21:30:36.726157 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] 8 chunks of 134217728 bytes to be transferred {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:421}}
Apr 09 21:30:37.901678 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 1 of 8 (111558K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:38.420803 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:30:38.424226 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:30:39.049865 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 2 of 8 (114215K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:40.154031 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 3 of 8 (118777K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:41.324972 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 4 of 8 (112002K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:42.647685 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 5 of 8 (99133K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:43.826361 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 6 of 8 (111260K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:44.965414 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 7 of 8 (115142K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:46.177547 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Transferred chunk 8 of 8 (108541K/s) {{(pid=134916) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}}
Apr 09 21:30:46.230331 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Getting metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Apr 09 21:30:46.242193 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134916) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Apr 09 21:30:46.242595 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Getting metadata type 'volume-metadata' {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Apr 09 21:30:46.249691 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] No metadata type 'volume-metadata' available {{(pid=134916) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Apr 09 21:30:46.250078 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Getting metadata type 'volume-glance-metadata' {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Apr 09 21:30:46.257395 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.driver [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] No metadata type 'volume-glance-metadata' available {{(pid=134916) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Apr 09 21:30:46.257945 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Backing up metadata for volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b. {{(pid=134916) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}}
Apr 09 21:30:46.290644 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Backup 'f41cb63e-911f-4569-881d-c775f1c4df59' of volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b finished. {{(pid=134916) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}}
Apr 09 21:30:46.292124 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f84ed893370>, 'connection_properties': {'name': 'volumes/volume-a5c65ac7-7c90-409b-a38a-0d7304ee17bf', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a5c65ac7-7c90-409b-a38a-0d7304ee17bf', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f84ed888b20>}, 'force': True, ': True}" {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}}
Apr 09 21:30:46.303424 ncdv-master-1 cinder-backup[134916]: DEBUG os_brick.initiator.connectors.rbd [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] <== disconnect_volume: return (10ms) None {{(pid=134916) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}}
Apr 09 21:30:46.380942 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-77bd54df-6b92-4765-add7-8ebbfccafd14 tempest-VolumesSnapshotTestJSON-286608301 None] Create backup finished. backup: f41cb63e-911f-4569-881d-c775f1c4df59.
Apr 09 21:30:46.914302 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Delete backup started, backup: f41cb63e-911f-4569-881d-c775f1c4df59.
Apr 09 21:30:46.916862 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Delete started for backup=f41cb63e-911f-4569-881d-c775f1c4df59 {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}}
Apr 09 21:30:46.917327 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Trying diff format basename='volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.f41cb63e-911f-4569-881d-c775f1c4df59' for backup base image of volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}}
Apr 09 21:30:46.949229 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Image 'volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.f41cb63e-911f-4569-881d-c775f1c4df59' not found - trying diff format name {{(pid=134916) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:693}}
Apr 09 21:30:46.970695 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Looking for snapshot of backup base 'volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.base' {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}}
Apr 09 21:30:46.971151 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Backup base 'volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.base' has no snapshots {{(pid=134916) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:955}}
Apr 09 21:30:46.971592 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] No backup snapshot to delete {{(pid=134916) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}}
Apr 09 21:30:46.980374 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Deleting backup base image='volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.base' of volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.
Apr 09 21:30:47.653190 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Base backup image='volume-fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b.backup.base' of volume fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b deleted. {{(pid=134916) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}}
Apr 09 21:30:47.699186 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.backup.drivers.ceph [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Delete of backup 'f41cb63e-911f-4569-881d-c775f1c4df59' for volume 'fbfeac4b-61b9-4faf-a5c0-5f9332a0b38b' finished. {{(pid=134916) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}}
Apr 09 21:30:47.705104 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_db.sqlalchemy.engines [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134916) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Apr 09 21:30:47.729841 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.quota [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Created reservations ['bf2616be-c512-47a0-923f-f229f43d2e21', 'c8fe989c-f4f7-4f52-9371-4c5b043e85da'] {{(pid=134916) reserve /opt/stack/cinder/cinder/quota.py:755}}
Apr 09 21:30:47.756140 ncdv-master-1 cinder-backup[134916]: INFO cinder.backup.manager [None req-317f4615-9817-4829-a01c-397a339806db tempest-VolumesSnapshotTestJSON-286608301 None] Delete backup finished, backup f41cb63e-911f-4569-881d-c775f1c4df59 deleted.
Apr 09 21:31:38.421015 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:31:38.421774 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:32:38.426637 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:32:38.427817 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:33:38.425390 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:33:38.426496 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:34:38.423455 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:34:38.424728 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:35:38.422128 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:35:38.422856 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:36:38.417868 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:36:38.418757 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:37:38.415307 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:37:38.416521 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:38:38.414133 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:38:38.414853 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:39:38.419721 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:39:38.420422 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:40:38.416911 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:40:38.418082 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:41:38.416453 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:41:38.417536 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:42:38.416332 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:42:38.416980 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:43:38.422562 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:43:38.423661 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:44:38.421456 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:44:38.422613 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:45:38.420800 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:45:38.421452 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:46:38.417349 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:46:38.420119 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:47:38.417941 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:47:38.419065 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:48:38.417277 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:48:38.418056 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:49:38.414327 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:49:38.415504 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:50:38.413652 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:50:38.414723 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:51:38.411274 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:51:38.412008 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:52:38.408419 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:52:38.409158 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:53:38.413377 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:53:38.414109 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:54:38.419392 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:54:38.420246 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:55:38.416569 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:55:38.417717 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:56:38.413445 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:56:38.414173 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:57:38.419232 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:57:38.419985 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:58:38.425556 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:58:38.426316 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 21:59:38.422448 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 21:59:38.423608 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:00:38.421513 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:00:38.422693 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:01:38.421174 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:01:38.422500 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:02:38.421118 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:02:38.422287 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:03:38.420102 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:03:38.420960 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:04:38.416531 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:04:38.417726 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:05:38.415727 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:05:38.416871 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:06:38.415931 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:06:38.417354 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:07:38.414840 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:07:38.415469 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:08:38.414329 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:08:38.415402 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:09:38.413494 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:09:38.414653 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:10:38.411986 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:10:38.412682 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:11:38.418166 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:11:38.419230 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:12:38.417231 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:12:38.417840 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:13:38.422635 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:13:38.423789 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:14:38.420903 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:14:38.422138 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:15:38.420579 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:15:38.421763 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:16:38.419446 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:16:38.420484 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:17:38.419366 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:17:38.420489 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:18:38.420761 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:18:38.421912 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:19:38.419481 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:19:38.420613 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Apr 09 22:20:38.426486 ncdv-master-1 cinder-backup[134916]: DEBUG oslo_service.periodic_task [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134916) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Apr 09 22:20:38.427205 ncdv-master-1 cinder-backup[134916]: DEBUG cinder.manager [None req-f99b1b4d-54ab-435e-a373-69e458c1708e None None] Notifying Schedulers of capabilities ... {{(pid=134916) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}