-- Logs begin at Sat 2021-11-06 20:01:16 UTC, end at Sat 2021-11-06 22:31:52 UTC. --
Nov 06 20:37:26.950235 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service.
Nov 06 20:37:30.192126 ncdv-master-1 cinder-backup[126950]: INFO cinder.cmd.backup [-] Backup running with 1 processes.
Nov 06 20:37:30.193868 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=126950) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}}
Nov 06 20:37:30.194421 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=126950) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}}
Nov 06 20:37:30.362261 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_db.sqlalchemy.engines [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=126950) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 20:37:30.607053 ncdv-master-1 cinder-backup[126950]: INFO cinder.rpc [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version.
Nov 06 20:37:30.611783 ncdv-master-1 cinder-backup[126950]: INFO cinder.rpc [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Automatically selected cinder-backup RPC version 2.2 as minimum service version.
Nov 06 20:37:30.616664 ncdv-master-1 cinder-backup[126950]: INFO cinder.rpc [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version.
Nov 06 20:37:30.621044 ncdv-master-1 cinder-backup[126950]: INFO cinder.rpc [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Automatically selected cinder-volume RPC version 3.16 as minimum service version.
Nov 06 20:37:30.626036 ncdv-master-1 cinder-backup[126950]: INFO cinder.rpc [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version.
Nov 06 20:37:30.633745 ncdv-master-1 cinder-backup[126950]: INFO cinder.rpc [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version.
Nov 06 20:37:30.691621 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_db.sqlalchemy.engines [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=126950) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 20:37:30.716542 ncdv-master-1 cinder-backup[126950]: INFO oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Starting 1 workers
Nov 06 20:37:30.722524 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Started child 127396 {{(pid=126950) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Nov 06 20:37:30.724175 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Full set of CONF: {{(pid=126950) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}}
Nov 06 20:37:30.724735 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] ******************************************************************************** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}}
Nov 06 20:37:30.725231 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] Configuration options gathered from: {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}}
Nov 06 20:37:30.725827 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}}
Nov 06 20:37:30.727091 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}}
Nov 06 20:37:30.727854 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] ================================================================================ {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}}
Nov 06 20:37:30.729079 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] allow_availability_zone_fallback = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.729676 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] allow_compression_on_image_upload = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.730424 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] allowed_direct_url_schemes     = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.730985 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] api_rate_limit                 = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.731600 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] auth_strategy                  = keystone {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.732195 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_availability_zone      = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.733063 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_api_class               = cinder.backup.api.API {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.733674 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_chunk_size         = 134217728 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.734250 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_conf               = /etc/ceph/ceph.conf {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.734860 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_image_journals     = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.735373 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_pool               = backups {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.735906 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_stripe_count       = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.736589 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_stripe_unit        = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.737110 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_ceph_user               = cinder-bak {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.737950 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_driver                  = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.738480 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_driver_init_check_interval = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.738989 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_driver_stats_polling_interval = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.739518 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_manager                 = cinder.backup.manager.BackupManager {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.740098 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_max_operations          = 15 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.740742 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_metadata_version        = 2 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.741284 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_name_template           = backup-%s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.741860 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_native_threads_pool_size = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.742405 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_object_number_per_notification = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.743176 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_service_inithost_offload = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.743718 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_timer_interval          = 120 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.744243 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_use_same_host           = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.744858 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_use_temp_snapshot       = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.745400 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backup_workers                 = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.745947 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] chap_password                  = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.746076 ncdv-master-1 cinder-backup[127396]: INFO cinder.service [-] Starting cinder-backup node (version 19.1.0)
Nov 06 20:37:30.747037 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] chap_username                  =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.747578 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] chiscsi_conf                   = /etc/chelsio-iscsi/chiscsi.conf {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.748164 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_internal_tenant_project_id = c964216bf9bf443e843de227ba6e6894 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.748734 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_internal_tenant_user_id = e1dc7c86c5a748f2b07331e648bbc704 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.749284 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] compression_format             = gzip {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.749863 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] config_dir                     = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.750362 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] config_file                    = ['/etc/cinder/cinder.conf'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.750908 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] config_source                  = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.751421 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] consistencygroup_api_class     = cinder.consistencygroup.api.API {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.751991 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] control_exchange               = openstack {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.752621 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] debug                          = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.753377 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] default_availability_zone      = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.753875 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] default_group_type             = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.754706 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.755279 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] default_volume_type            = ceph {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.756068 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] driver_client_cert             = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.756833 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] driver_client_cert_key         = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.758142 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] driver_data_namespace          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.758930 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] driver_ssl_cert_path           = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.759664 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] driver_ssl_cert_verify         = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.760497 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] driver_use_ssl                 = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.761203 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] enable_new_services            = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.761903 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] enable_unsupported_driver      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.762530 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] enabled_backends               = ['ceph'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.763340 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] enforce_multipath_for_image_xfer = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.763866 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] filter_function                = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.764381 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_api_insecure            = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.764961 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_api_servers             = ['http://10.222.0.13/image'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.765458 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_api_ssl_compression     = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.765951 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_ca_certificates_file    = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.766543 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_catalog_info            = image:glance:publicURL {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.767045 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_certfile                = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.767773 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_core_properties         = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.768280 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_keyfile                 = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.768992 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_num_retries             = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.769478 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] glance_request_timeout         = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.769985 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] goodness_function              = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.770513 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] graceful_shutdown_timeout      = 5 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.771021 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] group_api_class                = cinder.group.api.API {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.771539 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] host                           = ncdv-master-1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.772053 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iet_conf                       = /etc/iet/ietd.conf {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.772612 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_compress_on_upload       = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.773861 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_conversion_address_space_limit = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.774372 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_conversion_cpu_limit     = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.775113 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_conversion_dir           = /opt/stack/data/cinder/conversion {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.775622 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_upload_use_cinder_backend = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.776150 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_upload_use_internal_tenant = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.776693 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_volume_cache_enabled     = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.777280 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_volume_cache_max_count   = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.777788 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] image_volume_cache_max_size_gb = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.778312 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] instance_format                = [instance: %(uuid)s]  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.778836 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] instance_uuid_format           = [instance: %(uuid)s]  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.779386 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iscsi_iotype                   = fileio {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.779899 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iscsi_secondary_ip_addresses   = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.780521 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iscsi_target_flags             =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.781033 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iscsi_write_cache              = on {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.781535 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iser_helper                    = tgtadm {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.782321 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iser_ip_address                = 10.222.0.13 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.782870 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iser_port                      = 3260 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.783382 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] iser_target_prefix             = iqn.2010-10.org.openstack: {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.784794 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_config_append              = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.785308 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_date_format                = %Y-%m-%d %H:%M:%S {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.785826 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_dir                        = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.786550 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_file                       = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.786987 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_options                    = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.787485 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_rotate_interval            = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.788114 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_rotate_interval_type       = days {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.788698 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] log_rotation_type              = none {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.789156 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] logging_context_format_string  = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.789622 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] logging_debug_format_suffix    = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.790140 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] logging_default_format_string  = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.790741 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] logging_exception_prefix       = ERROR %(name)s %(instance)s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.791187 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] logging_user_identity_format   = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.791964 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] max_age                        = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.792522 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] max_logfile_count              = 30 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.793116 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] max_logfile_size_mb            = 200 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.793645 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] max_over_subscription_ratio    = 20.0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.794274 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] message_reap_interval          = 86400 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.794931 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] message_ttl                    = 2592000 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.795447 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] monkey_patch                   = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.795975 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] monkey_patch_modules           = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.796441 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] my_ip                          = 10.222.0.13 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.796936 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] no_snapshot_gb_quota           = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.797446 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] num_iser_scan_tries            = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.797965 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] num_shell_tries                = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.798465 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] num_volume_device_scan_tries   = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.798971 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] nvmet_ns_id                    = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.799464 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] nvmet_port_id                  = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.800003 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_max_limit                = 1000 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.800554 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_volume_ext_list          = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.801156 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_volume_extension         = ['cinder.api.contrib.standard_extensions'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.801676 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_volume_listen            = 0.0.0.0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.802200 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_volume_listen_port       = 8776 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.802702 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_volume_use_ssl           = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.803039 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.lockutils [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] Lock "cleanup_incomplete_backups_126950" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=127396) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}}
Nov 06 20:37:30.803361 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] Cleaning up incomplete backup operations.
Nov 06 20:37:30.803535 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] osapi_volume_workers           = 2 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.804846 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] per_volume_size_limit          = -1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.805347 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] periodic_fuzzy_delay           = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.805866 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] periodic_interval              = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.806367 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] public_endpoint                = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.807060 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] publish_errors                 = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.807575 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_backup_gigabytes         = 1000 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.808269 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_backups                  = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.808805 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_consistencygroups        = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.809458 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_driver                   = cinder.quota.DbQuotaDriver {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.809962 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_gigabytes                = 1000 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.810503 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_groups                   = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.811014 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_snapshots                = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.811561 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] quota_volumes                  = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.812049 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] rate_limit_burst               = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.812656 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] rate_limit_except_level        = CRITICAL {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.813173 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] rate_limit_interval            = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.813717 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] replication_device             = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.814417 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] report_discard_supported       = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.815156 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] report_interval                = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.815750 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] reservation_clean_interval     = 86400 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.816215 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] reservation_expire             = 86400 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.816813 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] reserved_percentage            = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.817352 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] resource_query_filters_file    = /etc/cinder/resource_filters.json {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.817878 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] restore_discard_excess_bytes   = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.818380 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] rootwrap_config                = /etc/cinder/rootwrap.conf {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.819090 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] rpc_response_timeout           = 120 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.819602 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] run_external_periodic_tasks    = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.820690 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] scheduler_manager              = cinder.scheduler.manager.SchedulerManager {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.821243 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] scst_target_driver             = iscsi {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.821751 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] scst_target_iqn_name           = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.822235 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_down_time              = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.822832 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] snapshot_name_template         = snapshot-%s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.823398 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] split_loggers                  = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.823835 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] state_path                     = /opt/stack/data/cinder {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.824557 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] storage_availability_zone      = nova {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.825390 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] storage_protocol               = iscsi {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.825934 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] syslog_log_facility            = LOG_USER {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.826484 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] target_helper                  = lioadm {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.827219 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] target_ip_address              = 10.222.0.13 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.827755 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] target_port                    = 3260 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.828575 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] target_prefix                  = iqn.2010-10.org.openstack: {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.829123 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] target_protocol                = iscsi {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.829636 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] trace_flags                    = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.830164 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] transfer_api_class             = cinder.transfer.api.API {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.830730 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] transport_url                  = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.831317 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] until_refresh                  = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.831810 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_chap_auth                  = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.832334 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_default_quota_class        = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.832898 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_eventlog                   = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.833412 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_journal                    = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.833917 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_json                       = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.834606 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_multipath_for_image_xfer   = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.835555 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_stderr                     = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.836093 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] use_syslog                     = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.836685 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] verify_glance_signatures       = enabled {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.837561 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_api_class               = cinder.volume.api.API {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.838404 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_backend_name            = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.838955 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_clear                   = zero {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.839507 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_clear_ionice            = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.840004 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_clear_size              = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.840536 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_copy_blkio_cgroup_name  = cinder-volume-copy {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.841076 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_copy_bps_limit          = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.841579 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_dd_blocksize            = 1M {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.842113 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_manager                 = cinder.volume.manager.VolumeManager {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.842618 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_name_template           = volume-%s {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.843212 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volume_usage_audit_period      = month {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.843821 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] volumes_dir                    = /opt/stack/data/cinder/volumes {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.844383 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] watch_log_file                 = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Nov 06 20:37:30.845441 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.846022 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.enforce_new_defaults = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.846637 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.enforce_scope      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.847217 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.policy_default_rule = default {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.847829 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.policy_dirs        = ['policy.d'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.848429 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.policy_file        = policy.yaml {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.848985 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.849532 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.850043 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.850542 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.851182 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.851721 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_concurrency.disable_process_locking = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.852240 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_concurrency.lock_path     = /opt/stack/data/cinder {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.852849 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.admin_password = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.853413 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.853918 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.admin_token = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.854418 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.admin_user  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.855095 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_admin_prefix =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.855894 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_host   = 127.0.0.1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.856453 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_port   = 35357 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.856990 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_protocol = https {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.857605 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_section = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.858113 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_type   = password {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.858639 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_uri    = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.859132 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.auth_version = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.859693 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.cache       = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.860230 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.cafile      = /opt/stack/data/ca-bundle.pem {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.860756 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.certfile    = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.861251 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.delay_auth_decision = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.861824 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.862325 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.http_connect_timeout = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.862827 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.863329 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.identity_uri = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.863881 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.include_service_catalog = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.864483 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.insecure    = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.865126 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.interface   = public {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.865877 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.keyfile     = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.866398 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.867509 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.868015 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.868593 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.869126 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.869706 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.870242 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.870743 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.871311 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.872364 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.region_name = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.872893 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.873412 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.service_token_roles_required = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.873934 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.service_type = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.874451 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.token_cache_time = 300 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.874976 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.878390 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.backend                  = dogpile.cache.null {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.878984 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.backend_argument         = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.879523 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.config_prefix            = cache.oslo {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.880064 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.debug_cache_backend      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.880606 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.enabled                  = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.881158 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.expiration_time          = 600 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.881679 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_dead_retry      = 300 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.882269 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.883290 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.883866 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_pool_maxsize    = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.884450 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.885028 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_servers         = ['localhost:11211'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.885709 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.memcache_socket_timeout  = 1.0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.886297 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.proxies                  = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.886826 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.tls_allowed_ciphers      = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.887317 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.tls_cafile               = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.887898 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.tls_certfile             = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.888674 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.tls_enabled              = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.889205 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cache.tls_keyfile              = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.889770 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] coordination.backend_url       = etcd3+http://10.222.0.13:2379 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.890344 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.backend               = sqlalchemy {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.890819 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.connection            = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.891339 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.connection_debug      = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.891854 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.connection_parameters =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.892389 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.connection_recycle_time = 3600 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.892890 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.connection_trace      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.893386 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.db_inc_retry_interval = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.893905 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.db_max_retries        = 20 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.894411 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.db_max_retry_interval = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.894911 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.db_retry_interval     = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.895509 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.max_overflow          = 50 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.896020 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.max_pool_size         = 5 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.896572 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.max_retries           = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.897072 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.mysql_enable_ndb      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.897590 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.mysql_sql_mode        = TRADITIONAL {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.898119 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.pool_timeout          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.898790 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.retry_interval        = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.899305 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.slave_connection      = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.899890 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.sqlite_synchronous    = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.900427 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] database.use_db_reconnect      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.901032 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.901550 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.902119 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_metrics.metrics_process_name =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.902648 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.903152 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.903722 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.connection_string     = messaging:// {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.904199 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.enabled               = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.904727 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.es_doc_type           = notification {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.905285 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.es_scroll_size        = 10000 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.905910 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.es_scroll_time        = 2m {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.906415 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.filter_error_trace    = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.906929 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.hmac_keys             = SECRET_KEY {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.907447 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.sentinel_service_name = mymaster {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.907963 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.socket_timeout        = 0.1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.908670 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] profiler.trace_sqlalchemy      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.909256 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] privsep_osbrick.capabilities   = [21] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.909915 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] privsep_osbrick.group          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.910407 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] privsep_osbrick.helper_command = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.910921 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] privsep_osbrick.logger_name    = oslo_privsep.daemon {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.911427 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.911952 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] privsep_osbrick.user           = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.912620 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_sys_admin.capabilities  = [0, 1, 2, 3, 12, 21] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.913110 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_sys_admin.group         = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.913661 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_sys_admin.helper_command = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.914178 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_sys_admin.logger_name   = oslo_privsep.daemon {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.914674 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.915161 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] cinder_sys_admin.user          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.915805 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.auth_section      = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.916330 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.auth_type         = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.917092 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.cafile            = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.917584 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.certfile          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.918271 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.collect_timing    = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.918940 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.insecure          = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.919445 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.keyfile           = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.919995 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.send_service_user_token = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.920539 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.split_loggers     = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.921029 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] service_user.timeout           = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.921582 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.backend_availability_zone = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.922110 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.chap_password = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.922615 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.chap_username =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.923140 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.chiscsi_conf  = /etc/chelsio-iscsi/chiscsi.conf {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.923655 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.924156 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.924671 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.driver_client_cert = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.925183 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.driver_client_cert_key = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.925698 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.driver_data_namespace = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.926392 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.926997 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.927506 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.driver_use_ssl = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.928013 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.enable_deferred_deletion = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.928773 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.enable_unsupported_driver = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.929812 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.930304 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.filter_function = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.930798 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.goodness_function = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.931302 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iet_conf      = /etc/iet/ietd.conf {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.931883 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.932499 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.933199 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.933740 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.934305 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.934859 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iscsi_iotype  = fileio {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.935373 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.935879 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iscsi_target_flags =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.936563 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iscsi_write_cache = on {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.937077 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iser_helper   = tgtadm {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.938098 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iser_ip_address = 10.222.0.13 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.938420 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iser_port     = 3260 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.939158 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.939695 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.940990 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.941513 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.num_shell_tries = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.942064 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.943209 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.nvmet_ns_id   = 10 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.943716 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.nvmet_port_id = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.944263 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.945380 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rados_connection_interval = 5 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.945884 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rados_connection_retries = 3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.946521 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_ceph_conf =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.947048 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.947546 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.948054 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.948599 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.949318 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_pool      = rbd {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.949835 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_secret_uuid = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.950533 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.951064 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.rbd_user      = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.951692 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.952235 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.replication_device = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.952783 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.report_discard_supported = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.953272 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.953760 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.reserved_percentage = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.954259 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.scst_target_driver = iscsi {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.954735 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.scst_target_iqn_name = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.955244 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.storage_protocol = iscsi {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.955741 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.target_helper = tgtadm {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.956523 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.target_ip_address = 10.222.0.13 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.957035 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.target_port   = 3260 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.957550 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.958099 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.target_protocol = iscsi {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.958591 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.trace_flags   = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.959325 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.unique_fqdn_network = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.959820 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.use_chap_auth = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.960726 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.961249 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_backend_name = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.961793 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_clear  = zero {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.962285 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_clear_ionice = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.962779 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_clear_size = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.963289 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.963835 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.964408 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.965025 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] backend_defaults.volumes_dir   = /opt/stack/data/cinder/volumes {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.965579 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_reports.file_event_handler = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.966399 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.967012 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_reports.log_dir           = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.967584 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.968125 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.968685 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.969362 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.969894 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.970407 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.970956 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.971451 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.971964 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.972493 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.973053 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.973578 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.974116 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.974616 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.975133 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.975689 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.976225 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.976903 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.977416 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.977942 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.978459 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.979117 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.ssl      = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.979824 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.ssl_ca_file =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.980532 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.ssl_cert_file =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.981043 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.ssl_key_file =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.981556 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_rabbit.ssl_version =  {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.982118 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_notifications.driver = [] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.982703 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_notifications.retry = -1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.983267 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.983761 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] oslo_messaging_notifications.transport_url = **** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.984378 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] key_manager.backend            = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.984945 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.auth_endpoint         = http://localhost/identity/v3 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.985455 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.barbican_api_version  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.985957 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.barbican_endpoint     = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.986500 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.barbican_endpoint_type = public {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.987106 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.barbican_region_name  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.987593 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.cafile                = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.988081 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.certfile              = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.988795 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.collect_timing        = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.989289 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.insecure              = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.989936 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.keyfile               = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.990432 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.number_of_retries     = 60 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.990938 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.retry_delay           = 1 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.991454 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.send_service_user_token = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.992541 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.split_loggers         = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.993030 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.timeout               = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.993523 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.verify_ssl            = True {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.994030 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican.verify_ssl_path       = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.994557 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.auth_section = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.995056 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.auth_type = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.995551 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.cafile   = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.996095 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.certfile = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.996651 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.collect_timing = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.997251 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.insecure = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.997759 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.keyfile  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.998369 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.split_loggers = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.998857 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] barbican_service_user.timeout  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:30.999372 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.approle_role_id          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.000041 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.approle_secret_id        = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.001290 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.cafile                   = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.001791 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.certfile                 = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.002690 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.collect_timing           = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.003362 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.insecure                 = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.003862 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.keyfile                  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.004425 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.kv_mountpoint            = secret {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.004928 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.kv_version               = 2 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.005434 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.namespace                = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.005934 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.root_token_id            = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.006427 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.split_loggers            = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.006931 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.ssl_ca_crt_file          = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.008167 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.timeout                  = None {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.008709 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.use_ssl                  = False {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.009469 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] vault.vault_url                = http://127.0.0.1:8200 {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Nov 06 20:37:31.010094 ncdv-master-1 cinder-backup[126950]: DEBUG oslo_service.service [None req-223fb8a8-c4d5-4dff-9203-bfb9234480a7 None None] ******************************************************************************** {{(pid=126950) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}}
Nov 06 20:37:31.047008 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 20:37:31.085985 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.lockutils [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] Lock "cleanup_incomplete_backups_126950" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.282s {{(pid=127396) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}}
Nov 06 20:37:31.094848 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 20:37:31.100655 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:37:31.128755 ncdv-master-1 cinder-backup[127396]: INFO cinder.keymgr.migration [None req-a18ee588-7cf6-484d-89d8-35606a0d44ea None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Nov 06 20:37:31.150723 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.service [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] Creating RPC server for service cinder-backup {{(pid=127396) start /opt/stack/cinder/cinder/service.py:231}}
Nov 06 20:37:31.158545 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.service [None req-388d1bf0-6ce4-4f7a-a0bf-808a642cdf1b None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=127396) start /opt/stack/cinder/cinder/service.py:237}}
Nov 06 20:38:54.216833 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:38:54.217567 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:39:54.221755 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:39:54.222414 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:40:54.219715 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:40:54.220387 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:41:54.221171 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:41:54.221841 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:42:54.225582 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:42:54.226259 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:43:54.230191 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:43:54.230863 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:44:54.235192 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:44:54.235930 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:45:54.239586 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:45:54.240745 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:46:54.237728 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:46:54.238940 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:47:54.236796 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:47:54.236796 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:48:54.233140 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:48:54.233867 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:49:54.238179 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:49:54.238903 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:50:54.234296 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:50:54.235022 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:51:54.239704 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:51:54.240464 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:52:54.246117 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:52:54.246771 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:53:54.252709 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:53:54.252709 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:54:54.253241 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:54:54.255019 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:55:54.256532 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:55:54.257261 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:56:54.260326 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:56:54.260895 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:57:54.257011 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:57:54.258128 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:58:54.260434 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:58:54.261129 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 20:59:54.265520 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 20:59:54.266219 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:00:54.262073 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:00:54.262715 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:01:54.262355 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:01:54.263359 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:02:54.264617 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:02:54.265204 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:03:54.261013 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:03:54.261685 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:05:45.272204 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Create backup started, backup: f35d0d25-156b-4c86-8d77-c7980f4e5221 volume: b42fdfd4-1349-458d-819b-36de88c750f8.
Nov 06 21:05:45.326431 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:05:45.336274 ncdv-master-1 cinder-backup[127396]: INFO oslo.privsep.daemon [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpvw69qjb6/privsep.sock']
Nov 06 21:05:45.440591 ncdv-master-1 sudo[154212]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpvw69qjb6/privsep.sock
Nov 06 21:05:45.452552 ncdv-master-1 sudo[154212]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 06 21:05:47.720272 ncdv-master-1 sudo[154212]: pam_unix(sudo:session): session closed for user root
Nov 06 21:05:47.724602 ncdv-master-1 cinder-backup[127396]: INFO oslo.privsep.daemon [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Spawned new privsep daemon via rootwrap
Nov 06 21:05:47.725372 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Accepted privsep connection to /tmp/tmpvw69qjb6/privsep.sock {{(pid=127396) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:363}}
Nov 06 21:05:47.726742 ncdv-master-1 cinder-backup[127396]: INFO oslo.privsep.daemon [-] privsep daemon starting
Nov 06 21:05:47.727018 ncdv-master-1 cinder-backup[127396]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
Nov 06 21:05:47.727356 ncdv-master-1 cinder-backup[127396]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
Nov 06 21:05:47.727615 ncdv-master-1 cinder-backup[127396]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 154221
Nov 06 21:05:47.731029 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[baea4c34-d5dc-4943-8dc6-437a6ce12114]: (2,) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.132684 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.181702 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.049s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:48.182104 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[c7ab78c9-654c-4dcb-843f-51dc5de09320]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.184467 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.204955 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[17f7ba4b-821b-4fb8-ab3f-b107b9354de3]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:05:48.226485 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:05:48.229188 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[17f7ba4b-821b-4fb8-ab3f-b107b9354de3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.232064 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.253745 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a53161fe-2867-463c-8bc3-c136a318f87b]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:05:48.256446 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:05:48.267990 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[a53161fe-2867-463c-8bc3-c136a318f87b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.270724 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.320994 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.050s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:48.321302 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[7285ec36-2e41-4f28-9771-051c9c70a26c]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.322563 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.393474 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.070s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:48.396606 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[c8f11f1e-c879-43dc-8e19-129d58183551]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.399985 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.413663 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.014s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:48.414046 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[2a60bf24-deb1-44a1-9be5-c4abe7556ce7]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:48.415104 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:48.463598 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] CMD "nvme version" returned: 0 in 0.048s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:48.469336 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] <== get_connector_properties: return (3140ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:05:49.738069 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:05:49.740242 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f53130>, 'connection_properties': {'name': 'volumes/volume-b42fdfd4-1349-458d-819b-36de88c750f8', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b42fdfd4-1349-458d-819b-36de88c750f8', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:05:49.746402 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:05:49.843808 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] <== connect_volume: return (101ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45f794f0>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:05:49.852741 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:05:49.897721 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Starting backup of volume='b42fdfd4-1349-458d-819b-36de88c750f8'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:05:49.899863 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Volume file is RBD: attempting optimized backup {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Nov 06 21:05:49.953679 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Creating base image 'volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' {{(pid=127396) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Nov 06 21:05:49.954154 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:05:51.839719 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Using --from-snap 'None' for incremental backup of volume b42fdfd4-1349-458d-819b-36de88c750f8. {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Nov 06 21:05:51.840387 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Creating backup snapshot='backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443' {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Nov 06 21:05:52.839473 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Performing differential transfer from 'volume-b42fdfd4-1349-458d-819b-36de88c750f8' to 'volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:05:52.840177 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_cxb7m6in --pool volumes volumes/volume-b42fdfd4-1349-458d-819b-36de88c750f8@backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:05:52.841276 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:05:53.912371 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Differential backup transfer completed in 1.0715s {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Nov 06 21:05:53.914852 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:05:53.941091 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:05:53.942477 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:05:53.958994 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:05:53.959748 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:05:53.972906 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] No metadata type 'volume-glance-metadata' available {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Nov 06 21:05:53.974737 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Backing up metadata for volume b42fdfd4-1349-458d-819b-36de88c750f8. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:05:54.049546 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Backup 'f35d0d25-156b-4c86-8d77-c7980f4e5221' of volume b42fdfd4-1349-458d-819b-36de88c750f8 finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:05:54.051762 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f53130>, 'connection_properties': {'name': 'volumes/volume-b42fdfd4-1349-458d-819b-36de88c750f8', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b42fdfd4-1349-458d-819b-36de88c750f8', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45f794f0>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:05:54.082784 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] <== disconnect_volume: return (29ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:05:54.166314 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-657fccb3-b486-491c-99aa-1d287a499f87 tempest-VolumesBackupsAdminTest-530643389 None] Create backup finished. backup: f35d0d25-156b-4c86-8d77-c7980f4e5221.
Nov 06 21:05:54.259152 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:05:54.259562 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:05:54.908066 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c6c524de-bf4a-455c-8dce-0e7a3e0e0459 tempest-VolumesBackupsAdminTest-1224710227 None] Export record started, backup: f35d0d25-156b-4c86-8d77-c7980f4e5221.
Nov 06 21:05:54.911472 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c6c524de-bf4a-455c-8dce-0e7a3e0e0459 tempest-VolumesBackupsAdminTest-1224710227 None] Export record finished, backup f35d0d25-156b-4c86-8d77-c7980f4e5221 exported.
Nov 06 21:05:55.138936 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-94cb6f0c-6ecf-427f-92cd-94fdda9113cc tempest-VolumesBackupsAdminTest-1224710227 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiNDlkOTg5MjItYWYyNi00NTJmLWFhMjUtNGI3NzVhMzBjYzZmIiwgInVzZXJfaWQiOiAiY2ZkYWY1OGMxOTg2NGJkZGI3ZjFiYmRhOWZjZjNhYTEiLCAicHJvamVjdF9pZCI6ICIwNTYyN2EyNzIxMTU0YzI0YWY4NGI4MDRhZmU1YTA4MSIsICJ2b2x1bWVfaWQiOiAiYjQyZmRmZDQtMTM0OS00NThkLTgxOWItMzZkZTg4Yzc1MGY4IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0zNzYzODU0NTgiLCAiZGlzcGxheV9kZXNjcmlwdGlvbiI6IG51bGwsICJzZXJ2aWNlX21ldGFkYXRhIjogIntcImJhc2VcIjogXCJ2b2x1bWUtYjQyZmRmZDQtMTM0OS00NThkLTgxOWItMzZkZTg4Yzc1MGY4LmJhY2t1cC5mMzVkMGQyNS0xNTZiLTRjODYtOGQ3Ny1jNzk4MGY0ZTUyMjFcIn0iLCAic2VydmljZSI6ICJjaW5kZXIuYmFja3VwLmRyaXZlcnMuY2VwaC5DZXBoQmFja3VwRHJpdmVyIiwgIm9iamVjdF9jb3VudCI6IDAsICJ0ZW1wX3ZvbHVtZV9pZCI6IG51bGwsICJ0ZW1wX3NuYXBzaG90X2lkIjogbnVsbCwgIm51bV9kZXBlbmRlbnRfYmFja3VwcyI6IDAsICJzbmFwc2hvdF9pZCI6IG51bGwsICJkYXRhX3RpbWVzdGFtcCI6ICIyMDIxLTExLTA2VDIxOjA1OjQ1WiIsICJyZXN0b3JlX3ZvbHVtZV9pZCI6IG51bGwsICJtZXRhZGF0YSI6IHt9LCAiZW5jcnlwdGlvbl9rZXlfaWQiOiBudWxsLCAiY3JlYXRlZF9hdCI6ICIyMDIxLTExLTA2VDIxOjA1OjQ1WiIsICJ1cGRhdGVkX2F0IjogIjIwMjEtMTEtMDZUMjE6MDU6NTRaIiwgImRlbGV0ZWRfYXQiOiBudWxsLCAiZGVsZXRlZCI6IGZhbHNlfQ==.
Nov 06 21:05:55.149077 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-94cb6f0c-6ecf-427f-92cd-94fdda9113cc tempest-VolumesBackupsAdminTest-1224710227 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:05:55.207748 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-94cb6f0c-6ecf-427f-92cd-94fdda9113cc tempest-VolumesBackupsAdminTest-1224710227 None] Import record id 49d98922-af26-452f-aa25-4b775a30cc6f metadata from driver finished.
Nov 06 21:05:57.621203 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Restore backup started, backup: f35d0d25-156b-4c86-8d77-c7980f4e5221 volume: ad7edb44-5c1d-4563-80e5-bf2f04d8b469.
Nov 06 21:05:57.634576 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:05:57.640598 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.653150 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.017s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:57.653476 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[da0ba16d-3a60-40c9-b4d4-eb109409a1b6]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:57.654870 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.669639 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9d9bf029-0350-4763-9f12-bb56742a431b]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:05:57.670198 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:05:57.671693 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d9bf029-0350-4763-9f12-bb56742a431b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:57.675496 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.693295 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[421f63ac-7e59-4d14-814b-8f6a0c94cd24]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:05:57.693684 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:05:57.695136 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[421f63ac-7e59-4d14-814b-8f6a0c94cd24]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:57.700714 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.715436 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:57.715765 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[ae46bd29-8ed0-463f-899a-d33752eed820]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:57.719255 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.727609 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:57.727959 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[6c4a8725-93dd-4ac8-a8b4-dffaa820f26a]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:57.729150 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.737689 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:57.738022 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[15b70c8a-077f-4467-9466-1418a95bf378]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:05:57.738837 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:05:57.756545 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] CMD "nvme version" returned: 0 in 0.017s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:05:57.757880 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] <== get_connector_properties: return (122ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:05:58.584646 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:05:58.586607 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45e93160>, 'connection_properties': {'name': 'volumes/volume-ad7edb44-5c1d-4563-80e5-bf2f04d8b469', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ad7edb44-5c1d-4563-80e5-bf2f04d8b469', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:05:58.587308 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:05:58.633065 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] <== connect_volume: return (45ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45c9fdc0>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:05:58.642210 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:05:58.663614 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Starting restore from Ceph backup=f35d0d25-156b-4c86-8d77-c7980f4e5221 to volume=volume-ad7edb44-5c1d-4563-80e5-bf2f04d8b469 {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}}
Nov 06 21:05:58.760653 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Looking for snapshot of backup base 'volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:05:58.761091 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Found snapshot 'backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:05:58.767310 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Volume file is RBD. {{(pid=127396) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}}
Nov 06 21:05:58.812047 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Attempting differential restore. {{(pid=127396) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}}
Nov 06 21:05:58.812670 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Attempting incremental restore from base='volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' snap='backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443' {{(pid=127396) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}}
Nov 06 21:05:58.813070 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Performing differential transfer from 'volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' to 'volume-ad7edb44-5c1d-4563-80e5-bf2f04d8b469' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:05:58.813498 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221@backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:05:58.813925 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_mvpeq9cb --pool volumes - volumes/volume-ad7edb44-5c1d-4563-80e5-bf2f04d8b469' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:05:59.689395 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Restore transfer completed in 0.8757s {{(pid=127396) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}}
Nov 06 21:05:59.690450 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}}
Nov 06 21:05:59.715747 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Restoring volume base metadata {{(pid=127396) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Nov 06 21:05:59.738140 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] No metadata of type 'volume-metadata' to restore {{(pid=127396) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Nov 06 21:05:59.738690 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=127396) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Nov 06 21:05:59.742188 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Restore to volume ad7edb44-5c1d-4563-80e5-bf2f04d8b469 finished successfully. {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}}
Nov 06 21:05:59.743878 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45e93160>, 'connection_properties': {'name': 'volumes/volume-ad7edb44-5c1d-4563-80e5-bf2f04d8b469', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ad7edb44-5c1d-4563-80e5-bf2f04d8b469', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45c9fdc0>}, 'force': True, 'ignore_errors': False}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:05:59.754250 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] <== disconnect_volume: return (9ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:05:59.808695 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.manager [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Encryption key ID for volume ad7edb44-5c1d-4563-80e5-bf2f04d8b469 already matches encryption key ID in backup f35d0d25-156b-4c86-8d77-c7980f4e5221. {{(pid=127396) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}}
Nov 06 21:05:59.856436 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-4993a93e-0b87-42d5-9d97-039b5b4dbaf0 tempest-VolumesBackupsAdminTest-530643389 None] Finished restoring backup f35d0d25-156b-4c86-8d77-c7980f4e5221 to volume ad7edb44-5c1d-4563-80e5-bf2f04d8b469.
Nov 06 21:06:01.019677 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Delete backup started, backup: 49d98922-af26-452f-aa25-4b775a30cc6f.
Nov 06 21:06:01.022762 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Delete started for backup=49d98922-af26-452f-aa25-4b775a30cc6f {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:06:01.023329 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Trying diff format basename='volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' for backup base image of volume b42fdfd4-1349-458d-819b-36de88c750f8. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:06:01.102886 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Looking for snapshot of backup base 'volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:06:01.103605 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Backup '49d98922-af26-452f-aa25-4b775a30cc6f' has no snapshot {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}}
Nov 06 21:06:01.104390 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] No backup snapshot to delete {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}}
Nov 06 21:06:01.113425 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Backup base image of volume b42fdfd4-1349-458d-819b-36de88c750f8 still has 1 snapshots so skipping base image delete.
Nov 06 21:06:01.160230 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Metadata backup object 'backup.49d98922-af26-452f-aa25-4b775a30cc6f.meta' not found - ignoring {{(pid=127396) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}}
Nov 06 21:06:01.163500 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Delete of backup '49d98922-af26-452f-aa25-4b775a30cc6f' for volume 'b42fdfd4-1349-458d-819b-36de88c750f8' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:06:01.173704 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:01.199081 ncdv-master-1 cinder-backup[127396]: WARNING cinder.quota [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota.
Nov 06 21:06:01.200030 ncdv-master-1 cinder-backup[127396]: WARNING cinder.quota [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota.
Nov 06 21:06:01.218860 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Created reservations ['adede945-4f66-4fe2-8b01-6c6875862660', 'dc80e435-b444-41c7-8fc0-2796dffcbb96'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:06:01.262742 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-901c559f-a632-4ae2-8559-62c5bc46892a tempest-VolumesBackupsAdminTest-1224710227 None] Delete backup finished, backup 49d98922-af26-452f-aa25-4b775a30cc6f deleted.
Nov 06 21:06:02.184574 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Delete backup started, backup: f35d0d25-156b-4c86-8d77-c7980f4e5221.
Nov 06 21:06:02.204544 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Delete started for backup=f35d0d25-156b-4c86-8d77-c7980f4e5221 {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:06:02.206166 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Trying diff format basename='volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' for backup base image of volume b42fdfd4-1349-458d-819b-36de88c750f8. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:06:02.316424 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Looking for snapshot of backup base 'volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:06:02.316850 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Found snapshot 'backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:06:02.317421 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Deleting backup snapshot='backup.f35d0d25-156b-4c86-8d77-c7980f4e5221.snap.1636232751.8397443' {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Nov 06 21:06:02.884900 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Deleting backup base image='volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' of volume b42fdfd4-1349-458d-819b-36de88c750f8.
Nov 06 21:06:03.142221 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Base backup image='volume-b42fdfd4-1349-458d-819b-36de88c750f8.backup.f35d0d25-156b-4c86-8d77-c7980f4e5221' of volume b42fdfd4-1349-458d-819b-36de88c750f8 deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:06:03.237286 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Delete of backup 'f35d0d25-156b-4c86-8d77-c7980f4e5221' for volume 'b42fdfd4-1349-458d-819b-36de88c750f8' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:06:03.244184 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:03.284283 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Created reservations ['7aa9cac3-6cae-49c3-a06c-a1edf7702d13', '50f59248-e688-49db-8616-705bd0534315'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:06:03.342706 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-ecdf2ad3-fe86-4607-ac47-af702d5deaef tempest-VolumesBackupsAdminTest-530643389 None] Delete backup finished, backup f35d0d25-156b-4c86-8d77-c7980f4e5221 deleted.
Nov 06 21:06:05.768023 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Create backup started, backup: 64691351-933b-4a2c-aa0d-7d46e69a47b6 volume: e40133c0-f602-49cc-9afc-5ba557a58961.
Nov 06 21:06:05.791054 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:05.799319 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.810188 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.014s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:05.810848 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[8175855e-3bae-4ab0-99ff-82a615f371fe]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:05.819501 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.821620 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[596d4070-8682-4b10-b25a-c07e172b6e10]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:06:05.821910 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:06:05.823465 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[596d4070-8682-4b10-b25a-c07e172b6e10]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:05.825657 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.836159 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[230bdd0d-5d51-4369-9678-47326e88ae58]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:06:05.836505 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:06:05.837946 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[230bdd0d-5d51-4369-9678-47326e88ae58]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:05.839676 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.853795 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.015s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:05.854182 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[d51c4ddc-a6f6-4154-a41f-73b22d28fbca]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:05.855479 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.879973 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.016s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:05.880250 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[1a73e45f-af0b-4246-add6-965b7d476ba4]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:05.885908 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.897169 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:05.906463 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[f1b64a70-e28d-4568-a308-ff7099cace28]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:05.906463 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:05.932152 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] CMD "nvme version" returned: 0 in 0.033s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:05.937714 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] <== get_connector_properties: return (143ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:06.835449 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:06:06.839069 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45e93160>, 'connection_properties': {'name': 'volumes/volume-e40133c0-f602-49cc-9afc-5ba557a58961', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e40133c0-f602-49cc-9afc-5ba557a58961', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:06.841476 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:06:06.916523 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] <== connect_volume: return (75ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45c83a00>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:06.925238 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:06.953376 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Starting backup of volume='e40133c0-f602-49cc-9afc-5ba557a58961'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:06:06.955896 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Volume file is RBD: attempting optimized backup {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Nov 06 21:06:07.009300 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Creating base image 'volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' {{(pid=127396) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Nov 06 21:06:07.009851 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:06:07.052395 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Using --from-snap 'None' for incremental backup of volume e40133c0-f602-49cc-9afc-5ba557a58961. {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Nov 06 21:06:07.052823 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Creating backup snapshot='backup.64691351-933b-4a2c-aa0d-7d46e69a47b6.snap.1636232767.0523365' {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Nov 06 21:06:07.881095 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Performing differential transfer from 'volume-e40133c0-f602-49cc-9afc-5ba557a58961' to 'volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:06:07.881674 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_syg37gdz --pool volumes volumes/volume-e40133c0-f602-49cc-9afc-5ba557a58961@backup.64691351-933b-4a2c-aa0d-7d46e69a47b6.snap.1636232767.0523365 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:06:07.881878 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:06:08.901243 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Differential backup transfer completed in 1.0188s {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Nov 06 21:06:08.902525 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:06:08.917762 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:06:08.918328 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:06:08.925121 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:06:08.925597 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:06:08.932441 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] No metadata type 'volume-glance-metadata' available {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Nov 06 21:06:08.933054 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Backing up metadata for volume e40133c0-f602-49cc-9afc-5ba557a58961. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:06:08.983946 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Backup '64691351-933b-4a2c-aa0d-7d46e69a47b6' of volume e40133c0-f602-49cc-9afc-5ba557a58961 finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:06:08.985543 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45e93160>, 'connection_properties': {'name': 'volumes/volume-e40133c0-f602-49cc-9afc-5ba557a58961', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e40133c0-f602-49cc-9afc-5ba557a58961', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45c83a00>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:09.009122 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] <== disconnect_volume: return (22ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:09.068141 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-92b7c63f-4c29-45ab-aec6-a49710472374 tempest-VolumesBackupsAdminTest-530643389 None] Create backup finished. backup: 64691351-933b-4a2c-aa0d-7d46e69a47b6.
Nov 06 21:06:10.053268 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c997d8e6-c700-4cfb-8fb3-b79adc43a01f tempest-VolumesBackupsAdminTest-1224710227 None] Reset backup status started, backup_id: 64691351-933b-4a2c-aa0d-7d46e69a47b6, status: error.
Nov 06 21:06:10.053933 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c997d8e6-c700-4cfb-8fb3-b79adc43a01f tempest-VolumesBackupsAdminTest-1224710227 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver.
Nov 06 21:06:10.158982 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Delete backup started, backup: 64691351-933b-4a2c-aa0d-7d46e69a47b6.
Nov 06 21:06:10.164127 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Delete started for backup=64691351-933b-4a2c-aa0d-7d46e69a47b6 {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:06:10.170197 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Trying diff format basename='volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' for backup base image of volume e40133c0-f602-49cc-9afc-5ba557a58961. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:06:10.264118 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Looking for snapshot of backup base 'volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:06:10.265080 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Found snapshot 'backup.64691351-933b-4a2c-aa0d-7d46e69a47b6.snap.1636232767.0523365' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:06:10.265298 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Deleting backup snapshot='backup.64691351-933b-4a2c-aa0d-7d46e69a47b6.snap.1636232767.0523365' {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Nov 06 21:06:10.980816 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Deleting backup base image='volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' of volume e40133c0-f602-49cc-9afc-5ba557a58961.
Nov 06 21:06:11.286427 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Base backup image='volume-e40133c0-f602-49cc-9afc-5ba557a58961.backup.64691351-933b-4a2c-aa0d-7d46e69a47b6' of volume e40133c0-f602-49cc-9afc-5ba557a58961 deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:06:11.367285 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-c06a5906-bc90-4450-b1ca-4db141b0fbe1 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:11.382572 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Delete of backup '64691351-933b-4a2c-aa0d-7d46e69a47b6' for volume 'e40133c0-f602-49cc-9afc-5ba557a58961' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:06:11.431072 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Created reservations ['a001f06a-ef50-4c6c-8bfd-c76bc1d0a17e', '2c642130-9f29-4e1a-bc05-47973ad7333f'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:06:11.468575 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-b5fffd44-454f-4bb8-b500-72e4ce22c093 tempest-VolumesBackupsAdminTest-530643389 None] Delete backup finished, backup 64691351-933b-4a2c-aa0d-7d46e69a47b6 deleted.
Nov 06 21:06:37.077237 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Create backup started, backup: 1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d volume: a1e216d8-3a6b-4fcb-a9c0-a136b001e711.
Nov 06 21:06:37.087206 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:37.088742 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.095856 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:37.096197 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[995b09b4-c2a0-4a40-85cd-99bda368a0ea]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:37.097489 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.102952 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[79631364-de76-4f89-88bb-f5cdf77943bc]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:06:37.103594 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:06:37.104482 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[79631364-de76-4f89-88bb-f5cdf77943bc]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:37.105166 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.113528 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7b6754f6-e1c1-4175-adef-f13102ec65f2]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:06:37.114557 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[7b6754f6-e1c1-4175-adef-f13102ec65f2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:37.116657 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.124646 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:37.125177 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[2766a908-5e3f-4f20-9b09-c12597dd7434]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:37.129213 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.135764 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:37.136310 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[8b16e286-1386-42a8-ae91-67241d5bd0c1]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:37.137431 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.144838 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:37.145097 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[35d673b8-a00f-484b-955f-1e13f6bba0dc]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:37.145938 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:37.170752 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] CMD "nvme version" returned: 0 in 0.024s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:37.172251 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] <== get_connector_properties: return (83ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:37.897169 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:06:37.898565 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f10100>, 'connection_properties': {'name': 'volumes/volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a1e216d8-3a6b-4fcb-a9c0-a136b001e711', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:37.899575 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:06:37.952153 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] <== connect_volume: return (53ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e453914f0>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:37.960308 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:37.979862 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Starting backup of volume='a1e216d8-3a6b-4fcb-a9c0-a136b001e711'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:06:37.980948 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Volume file is RBD: attempting optimized backup {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Nov 06 21:06:38.003988 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Creating base image 'volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' {{(pid=127396) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Nov 06 21:06:38.004656 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:06:38.044515 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Using --from-snap 'None' for incremental backup of volume a1e216d8-3a6b-4fcb-a9c0-a136b001e711. {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Nov 06 21:06:38.045015 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Creating backup snapshot='backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d.snap.1636232798.0445373' {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Nov 06 21:06:38.655894 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Performing differential transfer from 'volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711' to 'volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:06:38.656659 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_65vlndch --pool volumes volumes/volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711@backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d.snap.1636232798.0445373 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:06:38.656751 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:06:39.678289 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Differential backup transfer completed in 1.0221s {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Nov 06 21:06:39.678900 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:06:39.695760 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:06:39.696261 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:06:39.703459 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:06:39.703887 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:06:39.711262 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] No metadata type 'volume-glance-metadata' available {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Nov 06 21:06:39.712066 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Backing up metadata for volume a1e216d8-3a6b-4fcb-a9c0-a136b001e711. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:06:39.750458 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Backup '1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' of volume a1e216d8-3a6b-4fcb-a9c0-a136b001e711 finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:06:39.752634 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f10100>, 'connection_properties': {'name': 'volumes/volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a1e216d8-3a6b-4fcb-a9c0-a136b001e711', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e453914f0>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:39.766063 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] <== disconnect_volume: return (12ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:39.833049 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c847e5b1-7456-41d6-a90f-440720cf3299 tempest-VolumesBackupsTest-994108957 None] Create backup finished. backup: 1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d.
Nov 06 21:06:40.274088 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Delete backup started, backup: 1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d.
Nov 06 21:06:40.276553 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Delete started for backup=1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:06:40.277261 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Trying diff format basename='volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' for backup base image of volume a1e216d8-3a6b-4fcb-a9c0-a136b001e711. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:06:40.351662 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Looking for snapshot of backup base 'volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:06:40.352114 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Found snapshot 'backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d.snap.1636232798.0445373' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:06:40.352640 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Deleting backup snapshot='backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d.snap.1636232798.0445373' {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Nov 06 21:06:40.739221 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Deleting backup base image='volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' of volume a1e216d8-3a6b-4fcb-a9c0-a136b001e711.
Nov 06 21:06:40.940311 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Base backup image='volume-a1e216d8-3a6b-4fcb-a9c0-a136b001e711.backup.1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' of volume a1e216d8-3a6b-4fcb-a9c0-a136b001e711 deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:06:40.976011 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Delete of backup '1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d' for volume 'a1e216d8-3a6b-4fcb-a9c0-a136b001e711' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:06:40.982000 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:41.012207 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Created reservations ['7b9c4d33-deac-4706-b556-5761e937d561', 'd4112b83-7152-462b-a687-0e0f6665c665'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:06:41.038160 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-2b841196-64de-45bf-beb7-86854960578e tempest-VolumesBackupsTest-994108957 None] Delete backup finished, backup 1bf4b2b2-0b3e-4f23-8953-6357d9f1ef2d deleted.
Nov 06 21:06:54.260720 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:06:54.261352 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:06:54.465319 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Create backup started, backup: 6e0d8ef6-b58b-4407-be3f-0eb2c6006594 volume: d97c184c-739e-4519-89af-db6cea863c7f.
Nov 06 21:06:54.479540 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:54.481213 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.522035 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.040s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:54.523553 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[c9bd3bc3-3088-4690-b684-ad8fec676218]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:54.524931 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.535076 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ca51767d-c6ec-4cd9-8553-3c7e998039d6]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:06:54.536846 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:06:54.539007 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[ca51767d-c6ec-4cd9-8553-3c7e998039d6]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:54.543121 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.561718 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c8e881f0-1ce9-4777-9841-7eae0e33b948]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:06:54.564148 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:06:54.568521 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[c8e881f0-1ce9-4777-9841-7eae0e33b948]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:54.571004 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.596874 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.025s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:54.597160 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[70d6c021-5416-4687-96b5-484260dcc85f]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:54.604873 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.637283 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.031s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:54.637604 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[bd94190d-a593-4a83-9250-2894c8cf8e9e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:54.641831 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.664037 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.023s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:54.664362 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[ce85c2e4-c653-43e1-9699-c397ce7df430]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:06:54.665277 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:06:54.709825 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] CMD "nvme version" returned: 0 in 0.044s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:06:54.711023 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] <== get_connector_properties: return (230ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:56.126715 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:06:56.128789 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e4539c1c0>, 'connection_properties': {'name': 'volumes/volume-d97c184c-739e-4519-89af-db6cea863c7f', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd97c184c-739e-4519-89af-db6cea863c7f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:06:56.129993 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:06:56.268223 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] <== connect_volume: return (138ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e454001f0>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:06:56.286177 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:06:56.364652 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Starting backup of volume='d97c184c-739e-4519-89af-db6cea863c7f'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:06:56.367660 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Volume file is RBD: attempting optimized backup {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Nov 06 21:06:56.435466 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Creating base image 'volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' {{(pid=127396) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Nov 06 21:06:56.436287 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:06:56.524525 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Using --from-snap 'None' for incremental backup of volume d97c184c-739e-4519-89af-db6cea863c7f. {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Nov 06 21:06:56.525002 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Creating backup snapshot='backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447' {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Nov 06 21:06:56.895879 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Performing differential transfer from 'volume-d97c184c-739e-4519-89af-db6cea863c7f' to 'volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:06:56.896308 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_kk9haqir --pool volumes volumes/volume-d97c184c-739e-4519-89af-db6cea863c7f@backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:06:56.896925 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:07:06.738349 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Differential backup transfer completed in 9.8418s {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Nov 06 21:07:06.738969 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:07:06.755315 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:07:06.755783 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:07:06.763274 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:07:06.763683 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:07:06.772510 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}}
Nov 06 21:07:06.772510 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Backing up metadata for volume d97c184c-739e-4519-89af-db6cea863c7f. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:07:06.817656 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Backup '6e0d8ef6-b58b-4407-be3f-0eb2c6006594' of volume d97c184c-739e-4519-89af-db6cea863c7f finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:07:06.819773 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e4539c1c0>, 'connection_properties': {'name': 'volumes/volume-d97c184c-739e-4519-89af-db6cea863c7f', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd97c184c-739e-4519-89af-db6cea863c7f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e454001f0>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:06.843583 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] <== disconnect_volume: return (23ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:06.938948 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-5627d29a-36ff-4bc0-8a35-266db59eabd5 tempest-VolumesBackupsTest-994108957 None] Create backup finished. backup: 6e0d8ef6-b58b-4407-be3f-0eb2c6006594.
Nov 06 21:07:08.872728 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Restore backup started, backup: 6e0d8ef6-b58b-4407-be3f-0eb2c6006594 volume: d533a7a6-a637-4338-98d6-2b3ebad2d411.
Nov 06 21:07:08.888831 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:08.892973 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:08.918502 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.026s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:08.918795 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[526f2a18-6438-4c98-86cb-6b8c2763c0d1]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:08.924282 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:08.951178 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b5438570-0b74-4895-b410-62fab36b4889]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:07:08.951642 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:07:08.953610 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5438570-0b74-4895-b410-62fab36b4889]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:08.960278 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:08.975663 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ed59d542-5b6f-4b13-b8f7-3356d881f5a9]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:07:08.976424 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:07:08.978021 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[ed59d542-5b6f-4b13-b8f7-3356d881f5a9]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:08.980408 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:09.001954 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.020s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:09.002847 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[a15d3d26-6c36-4ac3-bf2a-aaccfb8b341b]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:09.004772 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:09.020521 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.016s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:09.022665 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[41e0a413-4240-4689-bc52-b57dc7590b69]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:09.025636 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:09.045483 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.020s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:09.046753 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[a4920f78-a80e-4e69-b5c6-cece884d176e]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:09.049540 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:09.081244 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] CMD "nvme version" returned: 0 in 0.031s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:09.082364 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] <== get_connector_properties: return (193ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:10.372199 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:07:10.374043 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45400f70>, 'connection_properties': {'name': 'volumes/volume-d533a7a6-a637-4338-98d6-2b3ebad2d411', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd533a7a6-a637-4338-98d6-2b3ebad2d411', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:10.375286 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:07:10.490008 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] <== connect_volume: return (114ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e453eb610>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:10.498833 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:07:10.539027 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Starting restore from Ceph backup=6e0d8ef6-b58b-4407-be3f-0eb2c6006594 to volume=volume-d533a7a6-a637-4338-98d6-2b3ebad2d411 {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}}
Nov 06 21:07:10.774570 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Looking for snapshot of backup base 'volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:07:10.775331 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Found snapshot 'backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:07:10.799060 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Volume file is RBD. {{(pid=127396) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}}
Nov 06 21:07:10.871423 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Attempting differential restore. {{(pid=127396) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}}
Nov 06 21:07:10.872057 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Attempting incremental restore from base='volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' snap='backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447' {{(pid=127396) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}}
Nov 06 21:07:10.872526 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Performing differential transfer from 'volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' to 'volume-d533a7a6-a637-4338-98d6-2b3ebad2d411' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:07:10.873006 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594@backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:07:10.873333 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd__qfgmtu5 --pool volumes - volumes/volume-d533a7a6-a637-4338-98d6-2b3ebad2d411' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:07:16.920673 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Restore transfer completed in 6.0477s {{(pid=127396) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}}
Nov 06 21:07:16.922617 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}}
Nov 06 21:07:16.982195 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Restoring volume base metadata {{(pid=127396) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Nov 06 21:07:17.013211 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] No metadata of type 'volume-metadata' to restore {{(pid=127396) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Nov 06 21:07:17.015814 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Restoring volume glance metadata {{(pid=127396) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}}
Nov 06 21:07:17.288828 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Restore to volume d533a7a6-a637-4338-98d6-2b3ebad2d411 finished successfully. {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}}
Nov 06 21:07:17.296074 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45400f70>, 'connection_properties': {'name': 'volumes/volume-d533a7a6-a637-4338-98d6-2b3ebad2d411', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd533a7a6-a637-4338-98d6-2b3ebad2d411', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e453eb610>}, 'force': True, 'ignore_errors': False}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:17.308264 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] <== disconnect_volume: return (10ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:17.364503 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.manager [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Encryption key ID for volume d533a7a6-a637-4338-98d6-2b3ebad2d411 already matches encryption key ID in backup 6e0d8ef6-b58b-4407-be3f-0eb2c6006594. {{(pid=127396) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}}
Nov 06 21:07:17.406914 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-a44f39ea-122a-4508-9ada-ae072802fd28 tempest-VolumesBackupsTest-994108957 None] Finished restoring backup 6e0d8ef6-b58b-4407-be3f-0eb2c6006594 to volume d533a7a6-a637-4338-98d6-2b3ebad2d411.
Nov 06 21:07:21.001683 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Delete backup started, backup: 6e0d8ef6-b58b-4407-be3f-0eb2c6006594.
Nov 06 21:07:21.006382 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Delete started for backup=6e0d8ef6-b58b-4407-be3f-0eb2c6006594 {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:07:21.007061 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Trying diff format basename='volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' for backup base image of volume d97c184c-739e-4519-89af-db6cea863c7f. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:07:21.137389 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Looking for snapshot of backup base 'volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:07:21.138054 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Found snapshot 'backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:07:21.138633 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Deleting backup snapshot='backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594.snap.1636232816.5245447' {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Nov 06 21:07:21.378742 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-c06a5906-bc90-4450-b1ca-4db141b0fbe1 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:07:21.920514 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Deleting backup base image='volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' of volume d97c184c-739e-4519-89af-db6cea863c7f.
Nov 06 21:07:22.964443 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Base backup image='volume-d97c184c-739e-4519-89af-db6cea863c7f.backup.6e0d8ef6-b58b-4407-be3f-0eb2c6006594' of volume d97c184c-739e-4519-89af-db6cea863c7f deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:07:23.054350 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Delete of backup '6e0d8ef6-b58b-4407-be3f-0eb2c6006594' for volume 'd97c184c-739e-4519-89af-db6cea863c7f' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:07:23.115436 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Created reservations ['e27f3abe-d048-4a97-8d5f-d0c41fcaed3e', '924b0d4e-e4d2-4bbe-a9c3-20a61c5c196b'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:07:23.190816 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-1a164938-188b-4ad3-904e-4f2b6f56bc68 tempest-VolumesBackupsTest-994108957 None] Delete backup finished, backup 6e0d8ef6-b58b-4407-be3f-0eb2c6006594 deleted.
Nov 06 21:07:43.457021 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Create backup started, backup: bb1b6d79-e889-42c4-93f0-36f2c9c540f0 volume: 0c88cbc9-1cec-436f-a0a3-3093c6540c96.
Nov 06 21:07:43.470229 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:43.472626 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.524670 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.052s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:43.525176 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[4c93eba1-3405-4c20-8abe-2d1881a427fe]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:43.529434 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.546492 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[bba3617e-092a-4d19-95cb-0ef34dff581d]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:07:43.547111 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:07:43.548660 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[bba3617e-092a-4d19-95cb-0ef34dff581d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:43.551961 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.572436 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7b5896ee-4246-4e20-9ac7-366aedc1fb4c]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:07:43.573014 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:07:43.575212 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[7b5896ee-4246-4e20-9ac7-366aedc1fb4c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:43.579469 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.603067 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.024s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:43.603484 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[2d6ef567-76f3-4288-9e8e-8e9665ae3da5]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:43.607454 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.630301 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.025s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:43.630704 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[4bec5205-9883-46a9-a2e1-4c131e8d7e71]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:43.632227 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.656481 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.023s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:43.657116 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[a1454503-0a13-4d60-840b-e31f9ed75906]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:07:43.658438 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:07:43.695734 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] CMD "nvme version" returned: 0 in 0.037s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:07:43.698733 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] <== get_connector_properties: return (226ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:45.114847 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:07:45.116313 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45cfac70>, 'connection_properties': {'name': 'volumes/volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0c88cbc9-1cec-436f-a0a3-3093c6540c96', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:45.118414 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:07:45.229187 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] <== connect_volume: return (110ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e4535fe80>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:45.239975 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:07:45.269618 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Starting backup of volume='0c88cbc9-1cec-436f-a0a3-3093c6540c96'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:07:45.270831 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Volume file is RBD: attempting optimized backup {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Nov 06 21:07:45.322183 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Creating base image 'volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' {{(pid=127396) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Nov 06 21:07:45.323216 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:07:45.402181 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Using --from-snap 'None' for incremental backup of volume 0c88cbc9-1cec-436f-a0a3-3093c6540c96. {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Nov 06 21:07:45.402601 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Creating backup snapshot='backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0.snap.1636232865.4022036' {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Nov 06 21:07:46.108565 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Performing differential transfer from 'volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96' to 'volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:07:46.108979 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_ovwjqjtu --pool volumes volumes/volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96@backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0.snap.1636232865.4022036 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:07:46.109382 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:07:47.147056 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Differential backup transfer completed in 1.0388s {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Nov 06 21:07:47.148315 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:07:47.170884 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:07:47.171301 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:07:47.178019 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:07:47.178413 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:07:47.185044 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] No metadata type 'volume-glance-metadata' available {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Nov 06 21:07:47.186275 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Backing up metadata for volume 0c88cbc9-1cec-436f-a0a3-3093c6540c96. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:07:47.234521 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Backup 'bb1b6d79-e889-42c4-93f0-36f2c9c540f0' of volume 0c88cbc9-1cec-436f-a0a3-3093c6540c96 finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:07:47.239862 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45cfac70>, 'connection_properties': {'name': 'volumes/volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0c88cbc9-1cec-436f-a0a3-3093c6540c96', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e4535fe80>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:07:47.276584 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] <== disconnect_volume: return (35ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:07:47.352763 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-76563416-0117-4243-bb09-fedbafa0f850 tempest-VolumesBackupsV39Test-1523267356 None] Create backup finished. backup: bb1b6d79-e889-42c4-93f0-36f2c9c540f0.
Nov 06 21:07:47.786759 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Delete backup started, backup: bb1b6d79-e889-42c4-93f0-36f2c9c540f0.
Nov 06 21:07:47.789244 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Delete started for backup=bb1b6d79-e889-42c4-93f0-36f2c9c540f0 {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:07:47.789661 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Trying diff format basename='volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' for backup base image of volume 0c88cbc9-1cec-436f-a0a3-3093c6540c96. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:07:47.903055 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Looking for snapshot of backup base 'volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:07:47.903454 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Found snapshot 'backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0.snap.1636232865.4022036' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:07:47.903849 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Deleting backup snapshot='backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0.snap.1636232865.4022036' {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Nov 06 21:07:48.204500 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Deleting backup base image='volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' of volume 0c88cbc9-1cec-436f-a0a3-3093c6540c96.
Nov 06 21:07:48.653783 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Base backup image='volume-0c88cbc9-1cec-436f-a0a3-3093c6540c96.backup.bb1b6d79-e889-42c4-93f0-36f2c9c540f0' of volume 0c88cbc9-1cec-436f-a0a3-3093c6540c96 deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:07:48.741290 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Delete of backup 'bb1b6d79-e889-42c4-93f0-36f2c9c540f0' for volume '0c88cbc9-1cec-436f-a0a3-3093c6540c96' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:07:48.752594 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:07:48.792917 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Created reservations ['d9265311-c3aa-47c7-8c30-5a25dcd86b1d', '32d1d47e-7f73-4881-a7ce-175ebd871109'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:07:48.890994 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-97238694-00f7-427f-9071-599ab011440d tempest-VolumesBackupsV39Test-1523267356 None] Delete backup finished, backup bb1b6d79-e889-42c4-93f0-36f2c9c540f0 deleted.
Nov 06 21:07:54.266117 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:07:54.266802 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:08:54.262526 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:08:54.263269 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:09:54.259371 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:09:54.260609 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:10:54.256728 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:10:54.258032 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:11:54.254025 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:11:54.255091 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:12:54.251816 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:12:54.253145 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:13:54.250120 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:13:54.251442 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:14:54.246797 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:14:54.247527 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:15:54.244546 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:15:54.245298 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:16:54.250031 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:16:54.250659 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:17:54.254608 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:17:54.255308 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:18:54.259607 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:18:54.260121 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:19:54.257118 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:19:54.257865 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:20:54.253947 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:20:54.254649 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:21:08.566898 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Create backup started, backup: 8304e281-f852-4364-8852-02793cebb7c5 volume: 864396ca-37f1-4fc1-a294-fbcb0fe5a63f.
Nov 06 21:21:08.586194 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:21:08.588111 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.615303 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.026s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:08.616370 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[bec6e876-91ee-4998-9cd4-a2b418fa1a95]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:08.618536 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.627774 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d32f8125-a564-4401-ab75-1ce2ad670e1a]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:21:08.629061 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:21:08.631191 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[d32f8125-a564-4401-ab75-1ce2ad670e1a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:08.635633 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.643125 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d5e8b4a9-a22b-43c3-9c9e-72faa85ae6df]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:21:08.643417 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:21:08.644757 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[d5e8b4a9-a22b-43c3-9c9e-72faa85ae6df]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:08.650016 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.676904 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.029s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:08.677501 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[0a3a4e0e-5de8-4cfd-bc33-621f00e05a7c]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:08.678888 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.697635 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.018s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:08.697960 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[4cc827fc-5204-4f45-87e1-3eb2b4333b59]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:08.700402 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.708875 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:08.709273 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[1a777137-3d35-43af-aebe-f81b3bdcc1fa]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:08.710219 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:08.737629 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] CMD "nvme version" returned: 0 in 0.027s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:08.738707 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] <== get_connector_properties: return (151ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:21:09.440270 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:21:09.441989 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f53fd0>, 'connection_properties': {'name': 'volumes/volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '864396ca-37f1-4fc1-a294-fbcb0fe5a63f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:21:09.443030 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:21:09.509385 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] <== connect_volume: return (66ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45cb78e0>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:21:09.516942 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:21:09.534744 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Starting backup of volume='864396ca-37f1-4fc1-a294-fbcb0fe5a63f'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:21:09.535738 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Volume file is RBD: attempting optimized backup {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Nov 06 21:21:09.561899 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Creating base image 'volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' {{(pid=127396) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Nov 06 21:21:09.562402 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:21:09.598354 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Using --from-snap 'None' for incremental backup of volume 864396ca-37f1-4fc1-a294-fbcb0fe5a63f. {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Nov 06 21:21:09.598873 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Creating backup snapshot='backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782' {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Nov 06 21:21:09.986028 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Performing differential transfer from 'volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f' to 'volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:21:09.986674 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_iy07y5li --pool volumes volumes/volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f@backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:21:09.987125 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:21:16.045609 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Differential backup transfer completed in 6.0593s {{(pid=127396) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Nov 06 21:21:16.046277 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:21:16.058421 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:21:16.058796 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:21:16.065379 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:21:16.065773 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:21:16.072998 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}}
Nov 06 21:21:16.073626 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Backing up metadata for volume 864396ca-37f1-4fc1-a294-fbcb0fe5a63f. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:21:16.111988 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Backup '8304e281-f852-4364-8852-02793cebb7c5' of volume 864396ca-37f1-4fc1-a294-fbcb0fe5a63f finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:21:16.113573 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f53fd0>, 'connection_properties': {'name': 'volumes/volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '864396ca-37f1-4fc1-a294-fbcb0fe5a63f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45cb78e0>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:21:16.142231 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] <== disconnect_volume: return (27ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:21:16.204833 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-964c3fe3-c09c-41e7-af21-f78faf17f9b9 tempest-TestVolumeBackupRestore-1683798703 None] Create backup finished. backup: 8304e281-f852-4364-8852-02793cebb7c5.
Nov 06 21:21:18.121224 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Restore backup started, backup: 8304e281-f852-4364-8852-02793cebb7c5 volume: ecc8927f-ce43-4e60-90d6-a786d9aee3e5.
Nov 06 21:21:18.135339 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:21:18.146627 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.154595 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.014s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:18.154887 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[ba705681-aa84-4c0b-9d59-9e6772d82c5a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:18.156242 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.162812 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0982441a-b4be-4dd3-8ab6-be008631ea51]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:21:18.163694 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:21:18.165251 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[0982441a-b4be-4dd3-8ab6-be008631ea51]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:18.165575 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.172177 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[57e2157d-660b-4714-a27c-bfe68ad8634a]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:21:18.172955 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:21:18.173989 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[57e2157d-660b-4714-a27c-bfe68ad8634a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:18.174668 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.183126 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:18.183467 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[2c713319-2d10-416a-a512-479ade935a3d]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:18.184645 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.193917 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:18.194535 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[2826293a-b587-477d-b1b4-9618b7c61f37]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:18.195747 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.203068 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:18.203395 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[26554ee2-f442-4297-9862-0fdc62129e42]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:21:18.204759 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:21:18.224530 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:21:18.226237 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] <== get_connector_properties: return (89ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:21:18.973063 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:21:18.974635 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f69430>, 'connection_properties': {'name': 'volumes/volume-ecc8927f-ce43-4e60-90d6-a786d9aee3e5', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ecc8927f-ce43-4e60-90d6-a786d9aee3e5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:21:18.975744 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:21:19.020522 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] <== connect_volume: return (45ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45cbd5b0>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:21:19.028884 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:21:19.039919 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Starting restore from Ceph backup=8304e281-f852-4364-8852-02793cebb7c5 to volume=volume-ecc8927f-ce43-4e60-90d6-a786d9aee3e5 {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}}
Nov 06 21:21:19.104044 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Looking for snapshot of backup base 'volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:21:19.104580 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Found snapshot 'backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:21:19.109866 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Volume file is RBD. {{(pid=127396) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}}
Nov 06 21:21:19.144663 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Attempting differential restore. {{(pid=127396) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}}
Nov 06 21:21:19.145297 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Attempting incremental restore from base='volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' snap='backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782' {{(pid=127396) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}}
Nov 06 21:21:19.145415 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Performing differential transfer from 'volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' to 'volume-ecc8927f-ce43-4e60-90d6-a786d9aee3e5' {{(pid=127396) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Nov 06 21:21:19.145804 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5@backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782 -' into... {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Nov 06 21:21:19.146177 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_z9e7gpui --pool volumes - volumes/volume-ecc8927f-ce43-4e60-90d6-a786d9aee3e5' {{(pid=127396) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Nov 06 21:21:22.492730 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Restore transfer completed in 3.3468s {{(pid=127396) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}}
Nov 06 21:21:22.493892 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}}
Nov 06 21:21:22.521500 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Restoring volume base metadata {{(pid=127396) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Nov 06 21:21:22.540851 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] No metadata of type 'volume-metadata' to restore {{(pid=127396) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Nov 06 21:21:22.541332 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Restoring volume glance metadata {{(pid=127396) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}}
Nov 06 21:21:22.684198 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Restore to volume ecc8927f-ce43-4e60-90d6-a786d9aee3e5 finished successfully. {{(pid=127396) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}}
Nov 06 21:21:22.686078 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45f69430>, 'connection_properties': {'name': 'volumes/volume-ecc8927f-ce43-4e60-90d6-a786d9aee3e5', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ecc8927f-ce43-4e60-90d6-a786d9aee3e5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e45cbd5b0>}, 'force': True, 'ignore_errors': False}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:21:22.697811 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] <== disconnect_volume: return (10ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:21:22.750217 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.manager [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Encryption key ID for volume ecc8927f-ce43-4e60-90d6-a786d9aee3e5 already matches encryption key ID in backup 8304e281-f852-4364-8852-02793cebb7c5. {{(pid=127396) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}}
Nov 06 21:21:22.783332 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-c404ec06-71ae-4f71-84f1-77679411947a tempest-TestVolumeBackupRestore-1683798703 None] Finished restoring backup 8304e281-f852-4364-8852-02793cebb7c5 to volume ecc8927f-ce43-4e60-90d6-a786d9aee3e5.
Nov 06 21:21:54.258584 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:21:54.259235 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:22:26.552732 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Delete backup started, backup: 8304e281-f852-4364-8852-02793cebb7c5.
Nov 06 21:22:26.554477 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Delete started for backup=8304e281-f852-4364-8852-02793cebb7c5 {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:22:26.555103 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Trying diff format basename='volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' for backup base image of volume 864396ca-37f1-4fc1-a294-fbcb0fe5a63f. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:22:26.631355 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Looking for snapshot of backup base 'volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:22:26.631992 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Found snapshot 'backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Nov 06 21:22:26.632564 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Deleting backup snapshot='backup.8304e281-f852-4364-8852-02793cebb7c5.snap.1636233669.5983782' {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Nov 06 21:22:26.824787 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Deleting backup base image='volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' of volume 864396ca-37f1-4fc1-a294-fbcb0fe5a63f.
Nov 06 21:22:27.514046 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Base backup image='volume-864396ca-37f1-4fc1-a294-fbcb0fe5a63f.backup.8304e281-f852-4364-8852-02793cebb7c5' of volume 864396ca-37f1-4fc1-a294-fbcb0fe5a63f deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:22:27.556139 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Delete of backup '8304e281-f852-4364-8852-02793cebb7c5' for volume '864396ca-37f1-4fc1-a294-fbcb0fe5a63f' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:22:27.562468 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:22:27.586133 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Created reservations ['58f68abc-20f0-402e-aff7-1354348eb890', '0ec7d770-19b9-4ff3-b166-31bd6ba1e712'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:22:27.612311 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-744bed75-d2b8-4836-9ec1-cf56d810a668 tempest-TestVolumeBackupRestore-1683798703 None] Delete backup finished, backup 8304e281-f852-4364-8852-02793cebb7c5 deleted.
Nov 06 21:22:54.263325 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:22:54.264190 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:23:54.268281 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:23:54.268986 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:24:54.273157 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:24:54.273814 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:25:54.278470 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:25:54.279180 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:26:54.276173 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:26:54.276944 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:26:55.351512 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Create backup started, backup: 24a1e461-e509-438b-9d02-a1adceefdee3 volume: 0ec0a7db-df57-43d7-aeab-fb28d412a109 snapshot: 1a7dfa96-4907-4c17-a512-30733f3300b1.
Nov 06 21:26:55.358709 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:26:55.360188 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.367193 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:26:55.367525 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[a2e0f970-c665-4569-afa1-16d2fc40ee7c]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:efbdfd1ba2be\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:26:55.368747 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.373864 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[78f17131-5959-433a-a94e-2880bf0933a3]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:26:55.374516 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:26:55.375580 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[78f17131-5959-433a-a94e-2880bf0933a3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:26:55.375940 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.381513 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[8d4dfe0b-fa4b-4db5-8a94-166f258a9952]: [Errno 2] No such file or directory: 'systool' {{(pid=154221) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]: Traceback (most recent call last):
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     ret = func(*f_args, **f_kwargs)
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     return func(*args, **kwargs)
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     return putils.execute(on_execute=on_execute,
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     obj = subprocess.Popen(cmd,
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     self._execute_child(args, executable, preexec_fn, close_fds,
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]:     raise child_exception_type(errno_num, err_msg, err_filename)
Nov 06 21:26:55.382180 ncdv-master-1 cinder-backup[127396]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Nov 06 21:26:55.383303 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[8d4dfe0b-fa4b-4db5-8a94-166f258a9952]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:26:55.383752 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.419337 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.035s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:26:55.419673 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[1b16ea0d-87ef-4cc4-abf1-644cf1c8fdde]: (4, ('/dev/sda1\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:26:55.420899 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.429960 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:26:55.430283 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[04ab965a-136e-4271-b770-01e91a9d038b]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:26:55.431328 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.437725 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=154221) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:26:55.438040 ncdv-master-1 cinder-backup[127396]: DEBUG oslo.privsep.daemon [-] privsep: reply[59fbf3b5-2725-4b23-b5ab-386cfb64440f]: (4, ('c51a2dcc-2070-8f40-9b24-2660eadfabbb\n', '')) {{(pid=154221) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Nov 06 21:26:55.439421 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Running cmd (subprocess): nvme version {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Nov 06 21:26:55.456087 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_concurrency.processutils [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] CMD "nvme version" returned: 0 in 0.017s {{(pid=127396) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Nov 06 21:26:55.457214 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.utils [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] <== get_connector_properties: return (98ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:efbdfd1ba2be', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c51a2dcc-2070-8f40-9b24-2660eadfabbb', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:799dbd0a-7a12-4da1-9688-38b7123a8165'} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:26:56.466199 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connector [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Factory for rbd on None {{(pid=127396) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Nov 06 21:26:56.467634 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45d4ce80>, 'connection_properties': {'name': 'volumes/volume-90d0a252-2217-48eb-8627-e9707383cf6c', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '90d0a252-2217-48eb-8627-e9707383cf6c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:26:56.468687 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.linuxrbd [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] opening connection to ceph cluster (timeout=-1). {{(pid=127396) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Nov 06 21:26:56.520226 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] <== connect_volume: return (52ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e4537e070>} {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:26:56.527214 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:26:56.545501 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Starting backup of volume='0ec0a7db-df57-43d7-aeab-fb28d412a109'. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Nov 06 21:26:56.566372 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=127396) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Nov 06 21:26:56.566768 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Creating backup base image='volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.base' for volume 0ec0a7db-df57-43d7-aeab-fb28d412a109. {{(pid=127396) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}}
Nov 06 21:26:56.596577 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Copying data from volume 0ec0a7db-df57-43d7-aeab-fb28d412a109. {{(pid=127396) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}}
Nov 06 21:26:56.611907 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferring data between 'volume-0ec0a7db-df57-43d7-aeab-fb28d412a109' and 'volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.base' {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}}
Nov 06 21:26:56.612321 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] 8 chunks of 134217728 bytes to be transferred {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}}
Nov 06 21:26:57.609186 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 1 of 8 (131579K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:26:58.542700 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 2 of 8 (140486K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:26:59.507621 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 3 of 8 (135922K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:27:00.513793 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 4 of 8 (130347K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:27:01.479337 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 5 of 8 (135833K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:27:02.439235 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 6 of 8 (136642K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:27:03.417633 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 7 of 8 (134050K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:27:04.407231 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Transferred chunk 8 of 8 (132533K/s) {{(pid=127396) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Nov 06 21:27:04.447522 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Getting metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Nov 06 21:27:04.458304 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=127396) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Nov 06 21:27:04.458713 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Getting metadata type 'volume-metadata' {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Nov 06 21:27:04.464676 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] No metadata type 'volume-metadata' available {{(pid=127396) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Nov 06 21:27:04.465069 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Getting metadata type 'volume-glance-metadata' {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Nov 06 21:27:04.470935 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.driver [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] No metadata type 'volume-glance-metadata' available {{(pid=127396) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Nov 06 21:27:04.471467 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Backing up metadata for volume 0ec0a7db-df57-43d7-aeab-fb28d412a109. {{(pid=127396) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Nov 06 21:27:04.500702 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Backup '24a1e461-e509-438b-9d02-a1adceefdee3' of volume 0ec0a7db-df57-43d7-aeab-fb28d412a109 finished. {{(pid=127396) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Nov 06 21:27:04.502633 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f8e45d4ce80>, 'connection_properties': {'name': 'volumes/volume-90d0a252-2217-48eb-8627-e9707383cf6c', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '90d0a252-2217-48eb-8627-e9707383cf6c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f8e4537e070>}, 'force': True, 'ignore_errors': True}" {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Nov 06 21:27:04.517848 ncdv-master-1 cinder-backup[127396]: DEBUG os_brick.initiator.connectors.rbd [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] <== disconnect_volume: return (14ms) None {{(pid=127396) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Nov 06 21:27:04.598669 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-adc260ce-7174-49c7-b5ff-57a12f75f964 tempest-VolumesSnapshotTestJSON-340105368 None] Create backup finished. backup: 24a1e461-e509-438b-9d02-a1adceefdee3.
Nov 06 21:27:05.738704 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Delete backup started, backup: 24a1e461-e509-438b-9d02-a1adceefdee3.
Nov 06 21:27:05.740434 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Delete started for backup=24a1e461-e509-438b-9d02-a1adceefdee3 {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Nov 06 21:27:05.740859 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Trying diff format basename='volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.24a1e461-e509-438b-9d02-a1adceefdee3' for backup base image of volume 0ec0a7db-df57-43d7-aeab-fb28d412a109. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Nov 06 21:27:05.766433 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Image 'volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.24a1e461-e509-438b-9d02-a1adceefdee3' not found - trying diff format name {{(pid=127396) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}}
Nov 06 21:27:05.783673 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Looking for snapshot of backup base 'volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.base' {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Nov 06 21:27:05.784057 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Backup base 'volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.base' has no snapshots {{(pid=127396) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}}
Nov 06 21:27:05.784473 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] No backup snapshot to delete {{(pid=127396) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}}
Nov 06 21:27:05.790283 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Deleting backup base image='volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.base' of volume 0ec0a7db-df57-43d7-aeab-fb28d412a109.
Nov 06 21:27:06.282149 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Base backup image='volume-0ec0a7db-df57-43d7-aeab-fb28d412a109.backup.base' of volume 0ec0a7db-df57-43d7-aeab-fb28d412a109 deleted. {{(pid=127396) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Nov 06 21:27:06.316270 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.backup.drivers.ceph [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Delete of backup '24a1e461-e509-438b-9d02-a1adceefdee3' for volume '0ec0a7db-df57-43d7-aeab-fb28d412a109' finished. {{(pid=127396) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Nov 06 21:27:06.322358 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_db.sqlalchemy.engines [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=127396) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Nov 06 21:27:06.344455 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.quota [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Created reservations ['b2ffe3f9-e695-4253-b32a-b6e0bee1ae57', 'af7bc03a-3730-42b5-bb0d-1c8e26a7ce1a'] {{(pid=127396) reserve /opt/stack/cinder/cinder/quota.py:755}}
Nov 06 21:27:06.367884 ncdv-master-1 cinder-backup[127396]: INFO cinder.backup.manager [None req-0b479bdf-8eef-4ac4-a831-1da029510dd1 tempest-VolumesSnapshotTestJSON-340105368 None] Delete backup finished, backup 24a1e461-e509-438b-9d02-a1adceefdee3 deleted.
Nov 06 21:27:54.272118 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:27:54.272832 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:28:54.277416 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:28:54.278079 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:29:54.274256 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:29:54.274836 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:30:54.279473 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:30:54.280091 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:31:54.284110 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:31:54.284834 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:32:54.288982 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:32:54.289645 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:33:54.285611 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:33:54.286298 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:34:54.281486 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:34:54.282146 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:35:54.286148 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:35:54.286854 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:36:54.286877 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:36:54.287602 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:37:54.284538 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:37:54.285289 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:38:54.290243 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:38:54.290917 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:39:54.295132 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:39:54.295822 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:40:54.299954 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:40:54.300706 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:41:54.305429 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:41:54.306107 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:42:54.310745 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:42:54.311401 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:43:54.316260 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:43:54.316971 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:44:54.322053 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:44:54.322739 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:45:54.327007 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:45:54.327727 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:46:54.332489 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:46:54.333168 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:47:54.337475 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:47:54.338212 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:48:54.334774 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:48:54.335474 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:49:54.339107 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:49:54.339792 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:50:54.335988 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:50:54.336723 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:51:54.340783 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:51:54.341774 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:52:54.341036 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:52:54.341752 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:53:54.345730 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:53:54.346443 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:54:54.350966 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:54:54.351649 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:55:54.356196 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:55:54.356953 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:56:54.353627 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:56:54.354378 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:57:54.358116 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:57:54.358855 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:58:54.355084 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:58:54.355713 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 21:59:54.351924 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 21:59:54.352673 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:00:54.357110 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:00:54.357808 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:01:54.362378 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:01:54.363066 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:02:54.367354 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:02:54.368039 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:03:54.364856 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:03:54.365546 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:04:54.370625 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:04:54.371309 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:05:54.375575 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:05:54.376268 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:06:54.380736 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:06:54.381374 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:07:54.378800 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:07:54.379493 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:09:54.385034 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:09:54.385698 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:10:54.389894 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:10:54.390633 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:11:54.394685 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:11:54.395349 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:12:54.390978 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:12:54.391704 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:13:54.388262 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:13:54.388902 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:14:54.385101 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:14:54.385806 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:15:54.391096 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:15:54.391867 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:16:54.388288 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:16:54.388969 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:17:54.393367 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:17:54.394085 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:18:54.398544 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:18:54.399240 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:19:54.403222 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:19:54.404286 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:20:54.400049 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:20:54.401058 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:21:54.396147 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:21:54.396893 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:22:54.401084 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:22:54.401820 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:23:54.405494 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:23:54.406158 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:24:54.409714 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:24:54.410259 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:25:54.414325 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:25:54.415001 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:26:54.420225 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:26:54.420904 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:27:54.424949 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:27:54.425592 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:28:54.430502 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:28:54.431171 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:29:54.435147 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:29:54.435800 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Nov 06 22:30:54.432055 ncdv-master-1 cinder-backup[127396]: DEBUG oslo_service.periodic_task [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=127396) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Nov 06 22:30:54.432750 ncdv-master-1 cinder-backup[127396]: DEBUG cinder.manager [None req-789d7597-7fa2-4f03-9746-a4b2a2ce12e4 None None] Notifying Schedulers of capabilities ... {{(pid=127396) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}