-- Logs begin at Sat 2021-12-18 20:01:14 UTC, end at Sat 2021-12-18 22:15:12 UTC. --
Dec 18 20:37:44.856575 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service.
Dec 18 20:37:48.166886 ncdv-master-1 cinder-backup[133615]: INFO cinder.cmd.backup [-] Backup running with 1 processes.
Dec 18 20:37:48.168628 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=133615) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}}
Dec 18 20:37:48.169191 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=133615) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}}
Dec 18 20:37:48.326716 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_db.sqlalchemy.engines [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133615) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:37:48.556615 ncdv-master-1 cinder-backup[133615]: INFO cinder.rpc [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version.
Dec 18 20:37:48.562307 ncdv-master-1 cinder-backup[133615]: INFO cinder.rpc [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Automatically selected cinder-backup RPC version 2.2 as minimum service version.
Dec 18 20:37:48.567493 ncdv-master-1 cinder-backup[133615]: INFO cinder.rpc [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version.
Dec 18 20:37:48.572659 ncdv-master-1 cinder-backup[133615]: INFO cinder.rpc [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Automatically selected cinder-volume RPC version 3.16 as minimum service version.
Dec 18 20:37:48.577776 ncdv-master-1 cinder-backup[133615]: INFO cinder.rpc [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version.
Dec 18 20:37:48.582411 ncdv-master-1 cinder-backup[133615]: INFO cinder.rpc [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version.
Dec 18 20:37:48.631377 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_db.sqlalchemy.engines [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133615) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:37:48.653447 ncdv-master-1 cinder-backup[133615]: INFO oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Starting 1 workers
Dec 18 20:37:48.659556 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Started child 134059 {{(pid=133615) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Dec 18 20:37:48.660671 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Full set of CONF: {{(pid=133615) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}}
Dec 18 20:37:48.661040 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] ******************************************************************************** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}}
Dec 18 20:37:48.661345 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] Configuration options gathered from: {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}}
Dec 18 20:37:48.661754 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}}
Dec 18 20:37:48.662446 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}}
Dec 18 20:37:48.662739 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] ================================================================================ {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}}
Dec 18 20:37:48.663486 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] allow_availability_zone_fallback = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.663920 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] allow_compression_on_image_upload = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.664484 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] allowed_direct_url_schemes     = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.664910 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] api_rate_limit                 = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.665380 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] auth_strategy                  = keystone {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.665766 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_availability_zone      = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.666175 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_api_class               = cinder.backup.api.API {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.666685 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_chunk_size         = 134217728 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.667106 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_conf               = /etc/ceph/ceph.conf {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.667621 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_image_journals     = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.668062 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_pool               = backups {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.668449 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_stripe_count       = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.668824 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_stripe_unit        = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.669208 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_ceph_user               = cinder-bak {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.669612 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_driver                  = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.669993 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_driver_init_check_interval = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.670396 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_driver_stats_polling_interval = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.670825 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_manager                 = cinder.backup.manager.BackupManager {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.671240 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_max_operations          = 15 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.671630 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_metadata_version        = 2 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.672073 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_name_template           = backup-%s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.672441 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_native_threads_pool_size = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.672815 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_object_number_per_notification = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.673188 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_service_inithost_offload = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.673549 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_timer_interval          = 120 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.674485 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_use_same_host           = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.674898 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_use_temp_snapshot       = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.675316 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backup_workers                 = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.675713 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] chap_password                  = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.676148 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] chap_username                  =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.676540 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] chiscsi_conf                   = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.676972 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_internal_tenant_project_id = 0dd4d15f423642fa8da604b758048203 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.677378 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_internal_tenant_user_id = 62bd53b4d64643d28ac7f4ea0ea45f90 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.677904 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] compression_format             = gzip {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.678364 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] config_dir                     = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.678694 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] config_file                    = ['/etc/cinder/cinder.conf'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.679112 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] config_source                  = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.679490 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] consistencygroup_api_class     = cinder.consistencygroup.api.API {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.679897 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] control_exchange               = openstack {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.680357 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] debug                          = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.680713 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] default_availability_zone      = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.681081 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] default_group_type             = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.682429 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.683019 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] default_volume_type            = ceph {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.683385 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] driver_client_cert             = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.683887 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] driver_client_cert_key         = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.684463 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] driver_data_namespace          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.685068 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] driver_ssl_cert_path           = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.685666 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] driver_ssl_cert_verify         = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.686277 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] driver_use_ssl                 = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.686783 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] enable_new_services            = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.687817 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] enable_unsupported_driver      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.688487 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] enabled_backends               = ['ceph'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.688926 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] enforce_multipath_for_image_xfer = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.689343 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] filter_function                = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.689527 ncdv-master-1 cinder-backup[134059]: INFO cinder.service [-] Starting cinder-backup node (version 19.1.0)
Dec 18 20:37:48.690630 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_api_insecure            = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.690725 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_api_servers             = ['http://10.222.0.13/image'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.692253 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_api_ssl_compression     = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.692253 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_ca_certificates_file    = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.692253 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_catalog_info            = image:glance:publicURL {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.692253 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_certfile                = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.692646 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_core_properties         = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.693011 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_keyfile                 = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.693355 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_num_retries             = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.693716 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] glance_request_timeout         = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.694068 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] goodness_function              = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.694545 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] graceful_shutdown_timeout      = 5 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.694901 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] group_api_class                = cinder.group.api.API {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.695274 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] host                           = ncdv-master-1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.695754 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iet_conf                       = /etc/iet/ietd.conf {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.695989 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_compress_on_upload       = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.696352 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_conversion_address_space_limit = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.696689 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_conversion_cpu_limit     = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.697193 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_conversion_dir           = /opt/stack/data/cinder/conversion {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.697562 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_upload_use_cinder_backend = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.698079 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_upload_use_internal_tenant = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.698487 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_volume_cache_enabled     = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.698827 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_volume_cache_max_count   = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.699222 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] image_volume_cache_max_size_gb = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.699602 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] instance_format                = [instance: %(uuid)s]  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.699979 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] instance_uuid_format           = [instance: %(uuid)s]  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.700436 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iscsi_iotype                   = fileio {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.700713 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iscsi_secondary_ip_addresses   = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.701091 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iscsi_target_flags             =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.701465 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iscsi_write_cache              = on {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.701824 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iser_helper                    = tgtadm {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.702531 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iser_ip_address                = 10.222.0.13 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.702970 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iser_port                      = 3260 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.703325 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] iser_target_prefix             = iqn.2010-10.org.openstack: {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.703677 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_config_append              = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.704729 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_date_format                = %Y-%m-%d %H:%M:%S {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.705075 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_dir                        = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.705441 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_file                       = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.705726 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_options                    = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.706084 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_rotate_interval            = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.706504 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_rotate_interval_type       = days {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.706865 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] log_rotation_type              = none {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.707203 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] logging_context_format_string  = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.707491 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] logging_debug_format_suffix    = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.707859 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] logging_default_format_string  = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.708411 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] logging_exception_prefix       = ERROR %(name)s %(instance)s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.708667 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] logging_user_identity_format   = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.709042 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] max_age                        = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.709402 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] max_logfile_count              = 30 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.709773 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] max_logfile_size_mb            = 200 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.710152 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] max_over_subscription_ratio    = 20.0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.710555 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] message_reap_interval          = 86400 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.710908 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] message_ttl                    = 2592000 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.711245 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] monkey_patch                   = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.711608 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] monkey_patch_modules           = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.711890 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] my_ip                          = 10.222.0.13 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.712353 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] no_snapshot_gb_quota           = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.712770 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] num_iser_scan_tries            = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.713218 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] num_shell_tries                = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.713566 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] num_volume_device_scan_tries   = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.714044 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] nvmet_ns_id                    = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.714440 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] nvmet_port_id                  = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.714820 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_max_limit                = 1000 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.715285 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_volume_ext_list          = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.715755 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_volume_extension         = ['cinder.api.contrib.standard_extensions'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.716133 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_volume_listen            = 0.0.0.0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.716534 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_volume_listen_port       = 8776 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.716884 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_volume_use_ssl           = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.717231 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] osapi_volume_workers           = 2 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.717640 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] per_volume_size_limit          = -1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.718008 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] periodic_fuzzy_delay           = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.718548 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] periodic_interval              = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.718883 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] public_endpoint                = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.719227 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] publish_errors                 = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.719652 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_backup_gigabytes         = 1000 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.720020 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_backups                  = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.720475 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_consistencygroups        = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.720916 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_driver                   = cinder.quota.DbQuotaDriver {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.721276 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_gigabytes                = 1000 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.721659 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_groups                   = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.722052 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_snapshots                = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.722442 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] quota_volumes                  = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.722799 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] rate_limit_burst               = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.723150 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] rate_limit_except_level        = CRITICAL {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.723509 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] rate_limit_interval            = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.723921 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] replication_device             = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.724258 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] report_discard_supported       = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.724595 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] report_interval                = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.725036 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] reservation_clean_interval     = 86400 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.725322 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] reservation_expire             = 86400 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.725710 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] reserved_percentage            = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.726933 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] resource_query_filters_file    = /etc/cinder/resource_filters.json {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.727206 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] restore_discard_excess_bytes   = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.727581 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] rootwrap_config                = /etc/cinder/rootwrap.conf {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.728018 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] rpc_response_timeout           = 120 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.728326 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] run_external_periodic_tasks    = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.728880 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] scheduler_manager              = cinder.scheduler.manager.SchedulerManager {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.729265 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] scst_target_driver             = iscsi {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.729605 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] scst_target_iqn_name           = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.729943 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_down_time              = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.730354 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] snapshot_name_template         = snapshot-%s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.730698 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] split_loggers                  = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.730989 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] state_path                     = /opt/stack/data/cinder {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.731351 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] storage_availability_zone      = nova {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.731700 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] storage_protocol               = iscsi {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.732090 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] syslog_log_facility            = LOG_USER {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.732414 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] target_helper                  = lioadm {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.732863 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] target_ip_address              = 10.222.0.13 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.733224 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] target_port                    = 3260 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.733583 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] target_prefix                  = iqn.2010-10.org.openstack: {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.733986 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] target_protocol                = iscsi {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.734409 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] trace_flags                    = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.734763 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] transfer_api_class             = cinder.transfer.api.API {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.735137 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] transport_url                  = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.735484 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] until_refresh                  = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.735819 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_chap_auth                  = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.736171 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_default_quota_class        = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.737030 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_eventlog                   = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.737366 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_journal                    = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.737701 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_json                       = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.738086 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_multipath_for_image_xfer   = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.738484 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_stderr                     = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.738965 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] use_syslog                     = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.739365 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] verify_glance_signatures       = enabled {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.739720 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_api_class               = cinder.volume.api.API {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.740052 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_backend_name            = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.740427 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_clear                   = zero {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.740759 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_clear_ionice            = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.741112 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_clear_size              = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.741462 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_copy_blkio_cgroup_name  = cinder-volume-copy {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.741796 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_copy_bps_limit          = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.742195 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_dd_blocksize            = 1M {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.742637 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_manager                 = cinder.volume.manager.VolumeManager {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.742964 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_name_template           = volume-%s {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.743331 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volume_usage_audit_period      = month {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.743755 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] volumes_dir                    = /opt/stack/data/cinder/volumes {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.744092 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] watch_log_file                 = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Dec 18 20:37:48.744577 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.744980 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.745370 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.enforce_new_defaults = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.745740 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.enforce_scope      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.746131 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.policy_default_rule = default {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.747855 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.policy_dirs        = ['policy.d'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.747855 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.policy_file        = policy.yaml {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.748666 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.748666 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.748825 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_concurrency.disable_process_locking = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_concurrency.lock_path     = /opt/stack/data/cinder {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.admin_password = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751831 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.admin_token = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.751920 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.admin_user  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.752820 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_admin_prefix =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.753154 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_host   = 127.0.0.1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.753519 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_port   = 35357 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.753957 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_protocol = https {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.754326 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_section = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.754721 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_type   = password {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.755091 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_uri    = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.755425 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.auth_version = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.755782 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.cache       = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.756154 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.cafile      = /opt/stack/data/ca-bundle.pem {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.756587 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.certfile    = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.756827 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.delay_auth_decision = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.757203 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.757540 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.http_connect_timeout = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.757878 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.758283 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.identity_uri = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.758645 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.include_service_catalog = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.759094 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.insecure    = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.759486 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.interface   = public {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.759823 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.keyfile     = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.760175 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.760536 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.760897 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.761218 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.761559 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.761889 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.762265 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.762629 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.763057 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.763401 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.region_name = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.763772 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.764130 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.service_token_roles_required = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.764467 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.service_type = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.764828 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.token_cache_time = 300 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.765188 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.765657 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.backend                  = dogpile.cache.null {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.766032 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.backend_argument         = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.766463 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.config_prefix            = cache.oslo {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.766796 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.lockutils [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] Lock "cleanup_incomplete_backups_133615" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.002s {{(pid=134059) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}}
Dec 18 20:37:48.766906 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.debug_cache_backend      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.767169 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.enabled                  = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.767595 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] Cleaning up incomplete backup operations.
Dec 18 20:37:48.768253 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.expiration_time          = 600 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.768621 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_dead_retry      = 300 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.768977 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.769495 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.769850 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_pool_maxsize    = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.770504 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.770888 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_servers         = ['localhost:11211'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.771295 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.memcache_socket_timeout  = 1.0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.771643 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.proxies                  = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.771984 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.tls_allowed_ciphers      = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.772317 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.tls_cafile               = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.772653 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.tls_certfile             = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.772998 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.tls_enabled              = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.773327 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cache.tls_keyfile              = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.773709 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] coordination.backend_url       = etcd3+http://10.222.0.13:2379 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.774117 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.backend               = sqlalchemy {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.774484 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.connection            = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.774855 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.connection_debug      = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.775213 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.connection_parameters =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.775550 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.connection_recycle_time = 3600 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.775920 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.connection_trace      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.776258 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.db_inc_retry_interval = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.776592 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.db_max_retries        = 20 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.776926 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.db_max_retry_interval = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.777274 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.db_retry_interval     = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.777615 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.max_overflow          = 50 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.777955 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.max_pool_size         = 5 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.778331 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.max_retries           = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.778675 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.mysql_enable_ndb      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.779027 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.mysql_sql_mode        = TRADITIONAL {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.779525 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.pool_timeout          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.779875 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.retry_interval        = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.780234 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.slave_connection      = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.780576 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.sqlite_synchronous    = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.780921 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] database.use_db_reconnect      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.781663 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.782016 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.782399 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_metrics.metrics_process_name =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.782755 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.783096 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.783782 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.connection_string     = messaging:// {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.784070 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.enabled               = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.784644 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.es_doc_type           = notification {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.784992 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.es_scroll_size        = 10000 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.785356 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.es_scroll_time        = 2m {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.785695 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.filter_error_trace    = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.786048 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.hmac_keys             = SECRET_KEY {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.786442 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.sentinel_service_name = mymaster {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.786788 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.socket_timeout        = 0.1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.787127 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] profiler.trace_sqlalchemy      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.787537 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] privsep_osbrick.capabilities   = [21] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.788620 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] privsep_osbrick.group          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.788987 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] privsep_osbrick.helper_command = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.789332 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] privsep_osbrick.logger_name    = oslo_privsep.daemon {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.789829 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.790196 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] privsep_osbrick.user           = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.790653 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_sys_admin.capabilities  = [0, 1, 2, 3, 12, 21] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.790987 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_sys_admin.group         = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.791319 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_sys_admin.helper_command = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.791669 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_sys_admin.logger_name   = oslo_privsep.daemon {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.792005 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.792351 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] cinder_sys_admin.user          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.792726 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.auth_section      = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.793061 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.auth_type         = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.793391 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.cafile            = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.793720 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.certfile          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.794077 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.collect_timing    = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.794461 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.insecure          = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.794792 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.keyfile           = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.795128 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.send_service_user_token = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.795485 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.split_loggers     = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.796059 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] service_user.timeout           = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.796203 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.backend_availability_zone = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.796551 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.chap_password = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.796902 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.chap_username =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.797256 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.chiscsi_conf  = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.797594 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.797951 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.798333 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.driver_client_cert = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.798667 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.driver_client_cert_key = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.799521 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.driver_data_namespace = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.799981 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.800325 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.800664 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.driver_use_ssl = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.801005 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.enable_deferred_deletion = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.801361 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.enable_unsupported_driver = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.801705 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.802033 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.filter_function = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.802408 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.goodness_function = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.802781 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iet_conf      = /etc/iet/ietd.conf {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.803136 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.803564 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.803809 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.804142 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.804471 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.804825 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iscsi_iotype  = fileio {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.805167 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.805513 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iscsi_target_flags =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.805864 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iscsi_write_cache = on {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.806212 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iser_helper   = tgtadm {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.806906 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iser_ip_address = 10.222.0.13 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.807212 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iser_port     = 3260 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.807565 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.807922 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.808263 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.809309 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.num_shell_tries = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.809662 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.810141 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.nvmet_ns_id   = 10 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.810512 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.nvmet_port_id = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.810868 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.811203 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rados_connection_interval = 5 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.811553 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rados_connection_retries = 3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.812156 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_ceph_conf =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.812517 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.812862 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.813206 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.813544 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.813898 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_pool      = rbd {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.814235 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_secret_uuid = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.814637 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.815481 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.rbd_user      = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.815830 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.816221 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.replication_device = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.816493 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.report_discard_supported = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.816829 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.817172 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.reserved_percentage = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.817528 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.scst_target_driver = iscsi {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.817863 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.scst_target_iqn_name = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.818214 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.storage_protocol = iscsi {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.818616 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.target_helper = tgtadm {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.819078 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.target_ip_address = 10.222.0.13 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.819414 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.target_port   = 3260 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.819764 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.820241 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.target_protocol = iscsi {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.820574 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.trace_flags   = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.820913 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.unique_fqdn_network = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.821370 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.use_chap_auth = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.821718 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.822053 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_backend_name = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.822487 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_clear  = zero {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.822828 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_clear_ionice = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.823166 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_clear_size = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.823539 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.823873 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.824228 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.824670 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] backend_defaults.volumes_dir   = /opt/stack/data/cinder/volumes {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.825039 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_reports.file_event_handler = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.825594 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.825783 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_reports.log_dir           = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.826212 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.826605 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.826972 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.827330 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.827677 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.828022 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.828361 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.828695 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.829030 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.829589 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.830513 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.830868 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.831260 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.831632 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.831974 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.832350 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.832687 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.833026 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.833362 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.833716 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.834057 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.834660 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.ssl      = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.835026 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.ssl_ca_file =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.835385 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.ssl_cert_file =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.835739 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.ssl_key_file =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.836118 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_rabbit.ssl_version =  {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.836515 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_notifications.driver = [] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.836897 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_notifications.retry = -1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.837289 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.837674 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] oslo_messaging_notifications.transport_url = **** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.838086 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] key_manager.backend            = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.838587 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.auth_endpoint         = http://localhost/identity/v3 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.838940 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.barbican_api_version  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.839287 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.barbican_endpoint     = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.839673 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.barbican_endpoint_type = public {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.840030 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.barbican_region_name  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.840364 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.cafile                = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.840826 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.certfile              = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.841202 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.collect_timing        = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.841535 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.insecure              = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.841871 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.keyfile               = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.842225 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.number_of_retries     = 60 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.842581 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.retry_delay           = 1 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.842915 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.send_service_user_token = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.843251 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.split_loggers         = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.843581 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.timeout               = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.843916 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.verify_ssl            = True {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.844245 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican.verify_ssl_path       = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.844594 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.auth_section = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.844930 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.auth_type = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.845258 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.cafile   = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.845585 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.certfile = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.846583 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.collect_timing = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.847163 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.insecure = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.847733 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.keyfile  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.848331 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.split_loggers = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.848924 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] barbican_service_user.timeout  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.849586 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.approle_role_id          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.850188 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.approle_secret_id        = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.852425 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.cafile                   = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.852425 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.certfile                 = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.852425 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.collect_timing           = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.852425 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.insecure                 = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.852866 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.keyfile                  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.852995 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.kv_mountpoint            = secret {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.853342 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.kv_version               = 2 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.853697 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.namespace                = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.854035 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.root_token_id            = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.854408 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.split_loggers            = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.854742 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.ssl_ca_crt_file          = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.855077 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.timeout                  = None {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.855412 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.use_ssl                  = False {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.855764 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] vault.vault_url                = http://127.0.0.1:8200 {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Dec 18 20:37:48.856045 ncdv-master-1 cinder-backup[133615]: DEBUG oslo_service.service [None req-9899d2d4-da0c-4fbf-9917-c62ba38fcfd4 None None] ******************************************************************************** {{(pid=133615) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}}
Dec 18 20:37:48.914144 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:37:48.962806 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.lockutils [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] Lock "cleanup_incomplete_backups_133615" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.197s {{(pid=134059) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}}
Dec 18 20:37:48.972986 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:37:48.979850 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:37:49.012926 ncdv-master-1 cinder-backup[134059]: INFO cinder.keymgr.migration [None req-8c5077b1-cb5c-4028-9cbf-716aa6e87c19 None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Dec 18 20:37:49.048561 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.service [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] Creating RPC server for service cinder-backup {{(pid=134059) start /opt/stack/cinder/cinder/service.py:231}}
Dec 18 20:37:49.058924 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.service [None req-7ac8c43e-68bb-45c7-9973-7483ab38677d None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=134059) start /opt/stack/cinder/cinder/service.py:237}}
Dec 18 20:39:34.097400 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:39:34.098121 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:40:34.093830 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:40:34.094534 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:41:34.099748 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:41:34.100404 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:42:34.104005 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:42:34.104681 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:43:34.109244 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:43:34.110309 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:44:34.104876 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:44:34.105539 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:45:34.110055 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:45:34.110773 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:46:34.107169 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:46:34.107789 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:47:34.108541 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:47:34.109172 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:48:34.109093 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:48:34.109717 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:49:34.105176 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:49:34.106498 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:50:34.105029 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:50:34.105766 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:51:34.100342 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:51:34.101003 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:52:34.097345 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:52:34.097947 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:53:34.093356 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:53:34.094652 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:54:34.097442 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:54:34.098239 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:55:34.092886 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:55:34.092886 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:56:34.090958 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:56:34.091916 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:57:34.091661 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:57:34.091661 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:57:48.797628 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Create backup started, backup: d157fedf-164b-4cfc-b135-2c6328dcac6d volume: 8222c671-bce9-4818-87da-9a29a84c1f34.
Dec 18 20:57:48.816403 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:57:48.824196 ncdv-master-1 cinder-backup[134059]: INFO oslo.privsep.daemon [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp260rl1vl/privsep.sock']
Dec 18 20:57:49.005430 ncdv-master-1 sudo[153436]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp260rl1vl/privsep.sock
Dec 18 20:57:49.018982 ncdv-master-1 sudo[153436]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 18 20:57:49.197684 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-4fbd5774-91f2-4288-b903-68b292e0cd1a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:57:53.822299 ncdv-master-1 sudo[153436]: pam_unix(sudo:session): session closed for user root
Dec 18 20:57:53.841468 ncdv-master-1 cinder-backup[134059]: INFO oslo.privsep.daemon [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Spawned new privsep daemon via rootwrap
Dec 18 20:57:53.841468 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Accepted privsep connection to /tmp/tmp260rl1vl/privsep.sock {{(pid=134059) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:363}}
Dec 18 20:57:53.842656 ncdv-master-1 cinder-backup[134059]: INFO oslo.privsep.daemon [-] privsep daemon starting
Dec 18 20:57:53.842656 ncdv-master-1 cinder-backup[134059]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
Dec 18 20:57:53.847335 ncdv-master-1 cinder-backup[134059]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
Dec 18 20:57:53.847605 ncdv-master-1 cinder-backup[134059]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 153537
Dec 18 20:57:53.858614 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[1f58ab2c-8d87-4ff4-acb8-5fe174eff39e]: (2,) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.016216 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.064276 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.048s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:57:55.065517 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[09e04812-79b3-49ec-9182-758967805753]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.067400 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.093042 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[06aa9af0-5e69-450d-baf6-36d26a114b84]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 20:57:55.113688 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 20:57:55.114748 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[06aa9af0-5e69-450d-baf6-36d26a114b84]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.116516 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[eb412ff9-e54f-421e-9cd6-f6d517afbf6f]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 20:57:55.140049 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 20:57:55.141871 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[eb412ff9-e54f-421e-9cd6-f6d517afbf6f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.142777 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.203904 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.061s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:57:55.204190 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[3929e18b-7c7b-4a99-9b85-36530ae06802]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.208443 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.288318 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.082s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:57:55.289185 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[052f047d-c09a-4060-9231-3f03b2749f06]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.292049 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.344164 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.053s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:57:55.344835 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[f8b1618c-0407-4ca2-ace2-b0685ad3c0e0]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:57:55.346228 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:57:55.418841 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] CMD "nvme version" returned: 0 in 0.072s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:57:55.446329 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] <== get_connector_properties: return (6628ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:57:57.350764 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 20:57:57.353819 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991c305e0>, 'connection_properties': {'name': 'volumes/volume-8222c671-bce9-4818-87da-9a29a84c1f34', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8222c671-bce9-4818-87da-9a29a84c1f34', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:57:57.355257 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 20:57:57.440682 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] <== connect_volume: return (84ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991bcdbe0>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:57:57.488153 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Starting backup of volume='8222c671-bce9-4818-87da-9a29a84c1f34'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 20:57:57.489647 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Volume file is RBD: attempting optimized backup {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Dec 18 20:57:57.571877 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Creating base image 'volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' {{(pid=134059) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Dec 18 20:57:57.571877 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 20:57:59.523818 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Using --from-snap 'None' for incremental backup of volume 8222c671-bce9-4818-87da-9a29a84c1f34. {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Dec 18 20:57:59.525031 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Creating backup snapshot='backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606' {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Dec 18 20:58:00.489665 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Performing differential transfer from 'volume-8222c671-bce9-4818-87da-9a29a84c1f34' to 'volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 20:58:00.490081 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd__lpsi2b2 --pool volumes volumes/volume-8222c671-bce9-4818-87da-9a29a84c1f34@backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 20:58:00.490500 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 20:58:01.607846 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Differential backup transfer completed in 1.1172s {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Dec 18 20:58:01.608497 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 20:58:01.629062 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 20:58:01.629476 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 20:58:01.651924 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 20:58:01.652312 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 20:58:01.671535 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] No metadata type 'volume-glance-metadata' available {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Dec 18 20:58:01.672101 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Backing up metadata for volume 8222c671-bce9-4818-87da-9a29a84c1f34. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 20:58:01.750102 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Backup 'd157fedf-164b-4cfc-b135-2c6328dcac6d' of volume 8222c671-bce9-4818-87da-9a29a84c1f34 finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 20:58:01.753847 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991c305e0>, 'connection_properties': {'name': 'volumes/volume-8222c671-bce9-4818-87da-9a29a84c1f34', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8222c671-bce9-4818-87da-9a29a84c1f34', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991bcdbe0>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:01.798569 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] <== disconnect_volume: return (43ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:01.957325 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-f7b0af25-d11b-4c55-ae1b-961f0c385dec tempest-VolumesBackupsAdminTest-1670197762 None] Create backup finished. backup: d157fedf-164b-4cfc-b135-2c6328dcac6d.
Dec 18 20:58:02.412787 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-284c7ce6-9304-4005-b42f-81a9ca079a63 tempest-VolumesBackupsAdminTest-39771139 None] Export record started, backup: d157fedf-164b-4cfc-b135-2c6328dcac6d.
Dec 18 20:58:02.414641 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-284c7ce6-9304-4005-b42f-81a9ca079a63 tempest-VolumesBackupsAdminTest-39771139 None] Export record finished, backup d157fedf-164b-4cfc-b135-2c6328dcac6d exported.
Dec 18 20:58:02.598760 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-e9b9a5d0-bc44-481e-a56e-a0c649e5e82a tempest-VolumesBackupsAdminTest-39771139 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiMDJlYzBjNTEtYTcwYS00MzM1LWJiZWMtOTE4YTcxNjk3MzAyIiwgInVzZXJfaWQiOiAiNTZlNmViMmNkMTc3NDc5MzkyMTdiOWEwM2UxY2ExYjUiLCAicHJvamVjdF9pZCI6ICI4MGE4NWI3Nzc1NDc0YTNlYTExYzk5MmU3MDdhYzVjYyIsICJ2b2x1bWVfaWQiOiAiODIyMmM2NzEtYmNlOS00ODE4LTg3ZGEtOWEyOWE4NGMxZjM0IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0yNzI2Njg0NzAiLCAiZGlzcGxheV9kZXNjcmlwdGlvbiI6IG51bGwsICJzZXJ2aWNlX21ldGFkYXRhIjogIntcImJhc2VcIjogXCJ2b2x1bWUtODIyMmM2NzEtYmNlOS00ODE4LTg3ZGEtOWEyOWE4NGMxZjM0LmJhY2t1cC5kMTU3ZmVkZi0xNjRiLTRjZmMtYjEzNS0yYzYzMjhkY2FjNmRcIn0iLCAic2VydmljZSI6ICJjaW5kZXIuYmFja3VwLmRyaXZlcnMuY2VwaC5DZXBoQmFja3VwRHJpdmVyIiwgIm9iamVjdF9jb3VudCI6IDAsICJ0ZW1wX3ZvbHVtZV9pZCI6IG51bGwsICJ0ZW1wX3NuYXBzaG90X2lkIjogbnVsbCwgIm51bV9kZXBlbmRlbnRfYmFja3VwcyI6IDAsICJzbmFwc2hvdF9pZCI6IG51bGwsICJkYXRhX3RpbWVzdGFtcCI6ICIyMDIxLTEyLTE4VDIwOjU3OjQ4WiIsICJyZXN0b3JlX3ZvbHVtZV9pZCI6IG51bGwsICJtZXRhZGF0YSI6IHt9LCAiZW5jcnlwdGlvbl9rZXlfaWQiOiBudWxsLCAiY3JlYXRlZF9hdCI6ICIyMDIxLTEyLTE4VDIwOjU3OjQ4WiIsICJ1cGRhdGVkX2F0IjogIjIwMjEtMTItMThUMjA6NTg6MDJaIiwgImRlbGV0ZWRfYXQiOiBudWxsLCAiZGVsZXRlZCI6IGZhbHNlfQ==.
Dec 18 20:58:02.618763 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-e9b9a5d0-bc44-481e-a56e-a0c649e5e82a tempest-VolumesBackupsAdminTest-39771139 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:58:02.672081 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-e9b9a5d0-bc44-481e-a56e-a0c649e5e82a tempest-VolumesBackupsAdminTest-39771139 None] Import record id 02ec0c51-a70a-4335-bbec-918a71697302 metadata from driver finished.
Dec 18 20:58:05.154094 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Restore backup started, backup: d157fedf-164b-4cfc-b135-2c6328dcac6d volume: 95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba.
Dec 18 20:58:05.205331 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:05.210948 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.245799 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.038s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:05.246879 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[bd56164a-4c26-40d4-8f57-522599e7e82a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:05.250941 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.272191 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1dd2d3b6-dcb8-4567-b1a8-1e80ac819e52]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 20:58:05.273457 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 20:58:05.274983 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[1dd2d3b6-dcb8-4567-b1a8-1e80ac819e52]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:05.276383 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.292119 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[02ca2e53-07a8-456c-90ab-8a1d0ccc9d8a]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 20:58:05.293046 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 20:58:05.295439 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[02ca2e53-07a8-456c-90ab-8a1d0ccc9d8a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:05.298758 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.323929 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.025s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:05.324451 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[f75acef1-0bad-4912-96e5-b55a9241e0d7]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:05.325757 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.363635 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.038s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:05.363963 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[c7d90f9c-ad01-4d83-9d06-6f3ba864da89]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:05.365138 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.395055 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.029s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:05.395608 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[16403ad7-89be-442a-9a4d-003fd2f8a9ad]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:05.396376 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:05.433037 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] CMD "nvme version" returned: 0 in 0.036s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:05.434847 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] <== get_connector_properties: return (228ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:06.476323 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 20:58:06.477766 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991d61ac0>, 'connection_properties': {'name': 'volumes/volume-95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:06.478970 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 20:58:06.570511 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] <== connect_volume: return (92ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991bf9100>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:06.578034 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:58:06.603774 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Starting restore from Ceph backup=d157fedf-164b-4cfc-b135-2c6328dcac6d to volume=volume-95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}}
Dec 18 20:58:06.727810 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Looking for snapshot of backup base 'volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 20:58:06.728343 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Found snapshot 'backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 20:58:06.735794 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Volume file is RBD. {{(pid=134059) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}}
Dec 18 20:58:06.802207 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Attempting differential restore. {{(pid=134059) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}}
Dec 18 20:58:06.803112 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Attempting incremental restore from base='volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' snap='backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606' {{(pid=134059) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}}
Dec 18 20:58:06.803675 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Performing differential transfer from 'volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' to 'volume-95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 20:58:06.804328 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d@backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 20:58:06.804860 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_kzu72aof --pool volumes - volumes/volume-95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 20:58:08.250491 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Restore transfer completed in 1.4459s {{(pid=134059) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}}
Dec 18 20:58:08.257184 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}}
Dec 18 20:58:08.315872 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Restoring volume base metadata {{(pid=134059) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Dec 18 20:58:08.340430 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] No metadata of type 'volume-metadata' to restore {{(pid=134059) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Dec 18 20:58:08.340862 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=134059) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Dec 18 20:58:08.344820 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Restore to volume 95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba finished successfully. {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}}
Dec 18 20:58:08.347597 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991d61ac0>, 'connection_properties': {'name': 'volumes/volume-95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991bf9100>}, 'force': True, 'ignore_errors': False}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:08.363683 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] <== disconnect_volume: return (15ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:08.421308 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.manager [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Encryption key ID for volume 95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba already matches encryption key ID in backup d157fedf-164b-4cfc-b135-2c6328dcac6d. {{(pid=134059) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}}
Dec 18 20:58:08.482830 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-12b4e21b-2aa3-4d14-b8c3-79cf2e61205d tempest-VolumesBackupsAdminTest-1670197762 None] Finished restoring backup d157fedf-164b-4cfc-b135-2c6328dcac6d to volume 95b27fc5-0e0a-4cee-b4ba-1e9a17fa53ba.
Dec 18 20:58:09.709375 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Delete backup started, backup: 02ec0c51-a70a-4335-bbec-918a71697302.
Dec 18 20:58:09.711632 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Delete started for backup=02ec0c51-a70a-4335-bbec-918a71697302 {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 20:58:09.714647 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Trying diff format basename='volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' for backup base image of volume 8222c671-bce9-4818-87da-9a29a84c1f34. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 20:58:09.782074 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Looking for snapshot of backup base 'volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 20:58:09.782474 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Backup '02ec0c51-a70a-4335-bbec-918a71697302' has no snapshot {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}}
Dec 18 20:58:09.782802 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] No backup snapshot to delete {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}}
Dec 18 20:58:09.791485 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Backup base image of volume 8222c671-bce9-4818-87da-9a29a84c1f34 still has 1 snapshots so skipping base image delete.
Dec 18 20:58:09.838965 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Metadata backup object 'backup.02ec0c51-a70a-4335-bbec-918a71697302.meta' not found - ignoring {{(pid=134059) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}}
Dec 18 20:58:09.842794 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Delete of backup '02ec0c51-a70a-4335-bbec-918a71697302' for volume '8222c671-bce9-4818-87da-9a29a84c1f34' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 20:58:09.850222 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:58:09.874414 ncdv-master-1 cinder-backup[134059]: WARNING cinder.quota [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota.
Dec 18 20:58:09.875665 ncdv-master-1 cinder-backup[134059]: WARNING cinder.quota [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota.
Dec 18 20:58:09.900105 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Created reservations ['2a7e31af-d222-44c2-9cb7-272a2c4f246e', '93bb25ec-61ec-485b-b024-3f606bbbd5f3'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 20:58:09.953858 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-7bf0bb3d-93df-43a7-9608-3fccdbbf7956 tempest-VolumesBackupsAdminTest-39771139 None] Delete backup finished, backup 02ec0c51-a70a-4335-bbec-918a71697302 deleted.
Dec 18 20:58:10.935499 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Delete backup started, backup: d157fedf-164b-4cfc-b135-2c6328dcac6d.
Dec 18 20:58:10.938549 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Delete started for backup=d157fedf-164b-4cfc-b135-2c6328dcac6d {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 20:58:10.939424 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Trying diff format basename='volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' for backup base image of volume 8222c671-bce9-4818-87da-9a29a84c1f34. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 20:58:11.016737 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Looking for snapshot of backup base 'volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 20:58:11.017449 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Found snapshot 'backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 20:58:11.018062 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Deleting backup snapshot='backup.d157fedf-164b-4cfc-b135-2c6328dcac6d.snap.1639861079.524606' {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Dec 18 20:58:11.190874 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Deleting backup base image='volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' of volume 8222c671-bce9-4818-87da-9a29a84c1f34.
Dec 18 20:58:11.406168 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Base backup image='volume-8222c671-bce9-4818-87da-9a29a84c1f34.backup.d157fedf-164b-4cfc-b135-2c6328dcac6d' of volume 8222c671-bce9-4818-87da-9a29a84c1f34 deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 20:58:11.455648 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Delete of backup 'd157fedf-164b-4cfc-b135-2c6328dcac6d' for volume '8222c671-bce9-4818-87da-9a29a84c1f34' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 20:58:11.462773 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:58:11.490605 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Created reservations ['b276fdf4-5827-404b-b143-079a7624f7cb', '2c7072b8-d277-419b-ac0c-0d088494112f'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 20:58:11.524132 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-b3a2e135-56ec-4bbc-96c4-eaba98657330 tempest-VolumesBackupsAdminTest-1670197762 None] Delete backup finished, backup d157fedf-164b-4cfc-b135-2c6328dcac6d deleted.
Dec 18 20:58:13.526411 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Create backup started, backup: ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e volume: 569eafdb-d840-4136-8d56-ebbc5607b8ab.
Dec 18 20:58:13.536709 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:13.540945 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.560478 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.022s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:13.560976 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[475793d2-64b9-4790-83ae-ea07750a3f9c]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:13.563304 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.573633 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a67be4a9-d1b6-4342-8414-1ab2e239cd55]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 20:58:13.574378 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 20:58:13.575324 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[a67be4a9-d1b6-4342-8414-1ab2e239cd55]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:13.576193 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.583551 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0320cfe1-ad14-47c7-8d3c-4f9a6491fa6c]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 20:58:13.584058 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 20:58:13.585014 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[0320cfe1-ad14-47c7-8d3c-4f9a6491fa6c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:13.595431 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.603969 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.016s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:13.604569 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[68c5d8bf-ce46-4ee4-864d-187221fb690e]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:13.606923 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.617234 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.011s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:13.617663 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[5e781af5-8811-4aff-86b9-084a11ceacf9]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:13.619036 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.638330 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.019s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:13.638961 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[e03313a3-c247-4067-817b-7168a2ade671]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 20:58:13.640679 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 20:58:13.687396 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] CMD "nvme version" returned: 0 in 0.046s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 20:58:13.688589 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] <== get_connector_properties: return (151ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:14.458656 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 20:58:14.460178 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991b53cd0>, 'connection_properties': {'name': 'volumes/volume-569eafdb-d840-4136-8d56-ebbc5607b8ab', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '569eafdb-d840-4136-8d56-ebbc5607b8ab', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:14.461623 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 20:58:14.563170 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] <== connect_volume: return (101ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991b64400>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:14.572486 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:58:14.596898 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Starting backup of volume='569eafdb-d840-4136-8d56-ebbc5607b8ab'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 20:58:14.597928 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Volume file is RBD: attempting optimized backup {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Dec 18 20:58:14.653390 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Creating base image 'volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' {{(pid=134059) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Dec 18 20:58:14.654602 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 20:58:14.727955 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Using --from-snap 'None' for incremental backup of volume 569eafdb-d840-4136-8d56-ebbc5607b8ab. {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Dec 18 20:58:14.728582 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Creating backup snapshot='backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e.snap.1639861094.7282007' {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Dec 18 20:58:15.245731 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Performing differential transfer from 'volume-569eafdb-d840-4136-8d56-ebbc5607b8ab' to 'volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 20:58:15.246531 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_u6nilzy_ --pool volumes volumes/volume-569eafdb-d840-4136-8d56-ebbc5607b8ab@backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e.snap.1639861094.7282007 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 20:58:15.247244 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 20:58:16.341786 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Differential backup transfer completed in 1.0955s {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Dec 18 20:58:16.342596 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 20:58:16.370408 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 20:58:16.370822 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 20:58:16.392294 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 20:58:16.392726 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 20:58:16.410391 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] No metadata type 'volume-glance-metadata' available {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Dec 18 20:58:16.411727 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Backing up metadata for volume 569eafdb-d840-4136-8d56-ebbc5607b8ab. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 20:58:16.500602 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Backup 'ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' of volume 569eafdb-d840-4136-8d56-ebbc5607b8ab finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 20:58:16.502149 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991b53cd0>, 'connection_properties': {'name': 'volumes/volume-569eafdb-d840-4136-8d56-ebbc5607b8ab', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '569eafdb-d840-4136-8d56-ebbc5607b8ab', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991b64400>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 20:58:16.546739 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] <== disconnect_volume: return (43ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 20:58:16.690636 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-0ce735fa-7119-497f-916a-8407bfc2b31f tempest-VolumesBackupsAdminTest-1670197762 None] Create backup finished. backup: ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e.
Dec 18 20:58:17.791186 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-efe64dc5-d10a-4f1c-9873-bb6769320cc6 tempest-VolumesBackupsAdminTest-39771139 None] Reset backup status started, backup_id: ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e, status: error.
Dec 18 20:58:17.792016 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-efe64dc5-d10a-4f1c-9873-bb6769320cc6 tempest-VolumesBackupsAdminTest-39771139 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver.
Dec 18 20:58:17.905122 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Delete backup started, backup: ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e.
Dec 18 20:58:17.906955 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Delete started for backup=ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 20:58:17.907383 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Trying diff format basename='volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' for backup base image of volume 569eafdb-d840-4136-8d56-ebbc5607b8ab. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 20:58:17.970424 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Looking for snapshot of backup base 'volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 20:58:17.971605 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Found snapshot 'backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e.snap.1639861094.7282007' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 20:58:17.972006 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Deleting backup snapshot='backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e.snap.1639861094.7282007' {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Dec 18 20:58:18.338353 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Deleting backup base image='volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' of volume 569eafdb-d840-4136-8d56-ebbc5607b8ab.
Dec 18 20:58:18.623979 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Base backup image='volume-569eafdb-d840-4136-8d56-ebbc5607b8ab.backup.ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' of volume 569eafdb-d840-4136-8d56-ebbc5607b8ab deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 20:58:18.690545 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Delete of backup 'ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e' for volume '569eafdb-d840-4136-8d56-ebbc5607b8ab' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 20:58:18.697453 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 20:58:18.744686 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Created reservations ['19d52ed9-7b5f-4756-8d57-8802c46e13e6', '5522e884-53a5-46fb-9a1b-10ba44454d9b'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 20:58:18.777053 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-785c5b42-1e8d-434f-bf0f-e9a512570e82 tempest-VolumesBackupsAdminTest-1670197762 None] Delete backup finished, backup ff5e08e4-11b7-45ac-a95e-aa8b7ea7a82e deleted.
Dec 18 20:58:34.091212 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:58:34.091212 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 20:59:34.092687 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 20:59:34.093420 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:00:34.098748 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:00:34.100478 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:01:34.097603 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:01:34.098213 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:02:34.098349 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:02:34.099732 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:03:34.099481 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:03:34.102241 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:04:34.098281 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:04:34.099762 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:05:34.102200 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:05:34.102789 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:06:34.101186 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:06:34.101765 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:07:34.111315 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:07:34.111906 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:08:34.113229 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:08:34.114361 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:09:07.370991 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Create backup started, backup: 5692d1f5-bc74-4de8-a364-e31971adadf5 volume: deaf7c96-01b6-479a-946e-c92f2d37075d.
Dec 18 21:09:07.384140 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:09:07.391089 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.399951 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.014s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:09:07.403037 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[a73db6a7-6dc7-4396-9da6-0b119e953626]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:09:07.405690 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.414660 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[dd66af24-bfbc-41f0-9731-9b710034fc4f]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:09:07.415318 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:09:07.416561 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[dd66af24-bfbc-41f0-9731-9b710034fc4f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:09:07.417978 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.430112 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[60f5f077-224a-4f55-bea8-51bf69ad7b88]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:09:07.431378 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:09:07.432427 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[60f5f077-224a-4f55-bea8-51bf69ad7b88]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:09:07.433595 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.453109 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.019s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:09:07.453440 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[75ce00a3-c95b-455e-b018-897b72a564a9]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:09:07.454582 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.465882 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.011s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:09:07.466128 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[aeaf27c9-5e86-4f31-9ade-d8b08d7ce5bd]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:09:07.469664 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.482276 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.014s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:09:07.483122 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[619e6179-a88c-4cc3-b056-1a2a2c589f84]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:09:07.483985 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:09:07.513935 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] CMD "nvme version" returned: 0 in 0.030s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:09:07.515057 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] <== get_connector_properties: return (129ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:09:08.389913 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:09:08.391626 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991bafc40>, 'connection_properties': {'name': 'volumes/volume-deaf7c96-01b6-479a-946e-c92f2d37075d', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'deaf7c96-01b6-479a-946e-c92f2d37075d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:09:08.392716 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:09:08.515109 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] <== connect_volume: return (122ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991c11f10>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:09:08.523171 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:09:08.557722 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Starting backup of volume='deaf7c96-01b6-479a-946e-c92f2d37075d'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 21:09:08.559563 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Volume file is RBD: attempting optimized backup {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Dec 18 21:09:08.610434 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Creating base image 'volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' {{(pid=134059) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Dec 18 21:09:08.611009 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 21:09:08.708414 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Using --from-snap 'None' for incremental backup of volume deaf7c96-01b6-479a-946e-c92f2d37075d. {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Dec 18 21:09:08.708837 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Creating backup snapshot='backup.5692d1f5-bc74-4de8-a364-e31971adadf5.snap.1639861748.708438' {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Dec 18 21:09:09.257534 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Performing differential transfer from 'volume-deaf7c96-01b6-479a-946e-c92f2d37075d' to 'volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 21:09:09.258144 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_3998ck84 --pool volumes volumes/volume-deaf7c96-01b6-479a-946e-c92f2d37075d@backup.5692d1f5-bc74-4de8-a364-e31971adadf5.snap.1639861748.708438 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 21:09:09.258705 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 21:09:10.199626 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Differential backup transfer completed in 0.9418s {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Dec 18 21:09:10.200345 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 21:09:10.222356 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 21:09:10.222800 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 21:09:10.238180 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 21:09:10.239017 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 21:09:10.249375 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] No metadata type 'volume-glance-metadata' available {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Dec 18 21:09:10.250632 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Backing up metadata for volume deaf7c96-01b6-479a-946e-c92f2d37075d. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 21:09:10.355294 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Backup '5692d1f5-bc74-4de8-a364-e31971adadf5' of volume deaf7c96-01b6-479a-946e-c92f2d37075d finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 21:09:10.356957 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991bafc40>, 'connection_properties': {'name': 'volumes/volume-deaf7c96-01b6-479a-946e-c92f2d37075d', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'deaf7c96-01b6-479a-946e-c92f2d37075d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991c11f10>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:09:10.406505 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] <== disconnect_volume: return (47ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:09:10.588070 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-c88ca725-cd71-4d1e-a84e-2bf6e868e2ab tempest-VolumesBackupsV39Test-1873090783 None] Create backup finished. backup: 5692d1f5-bc74-4de8-a364-e31971adadf5.
Dec 18 21:09:11.687515 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Delete backup started, backup: 5692d1f5-bc74-4de8-a364-e31971adadf5.
Dec 18 21:09:11.690496 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Delete started for backup=5692d1f5-bc74-4de8-a364-e31971adadf5 {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 21:09:11.691195 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Trying diff format basename='volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' for backup base image of volume deaf7c96-01b6-479a-946e-c92f2d37075d. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 21:09:11.841254 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Looking for snapshot of backup base 'volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:09:11.841929 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Found snapshot 'backup.5692d1f5-bc74-4de8-a364-e31971adadf5.snap.1639861748.708438' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 21:09:11.842515 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Deleting backup snapshot='backup.5692d1f5-bc74-4de8-a364-e31971adadf5.snap.1639861748.708438' {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Dec 18 21:09:12.463660 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Deleting backup base image='volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' of volume deaf7c96-01b6-479a-946e-c92f2d37075d.
Dec 18 21:09:12.988659 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Base backup image='volume-deaf7c96-01b6-479a-946e-c92f2d37075d.backup.5692d1f5-bc74-4de8-a364-e31971adadf5' of volume deaf7c96-01b6-479a-946e-c92f2d37075d deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 21:09:13.137586 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Delete of backup '5692d1f5-bc74-4de8-a364-e31971adadf5' for volume 'deaf7c96-01b6-479a-946e-c92f2d37075d' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 21:09:13.148798 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:09:13.219134 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Created reservations ['eb849753-e872-47d5-ac35-967d1f033ef3', '246ce813-2a55-454f-8a6c-04ec00ea5309'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 21:09:13.294863 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-18e5f58b-277f-4484-abc3-494473539be4 tempest-VolumesBackupsV39Test-1873090783 None] Delete backup finished, backup 5692d1f5-bc74-4de8-a364-e31971adadf5 deleted.
Dec 18 21:09:34.113662 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:09:34.114367 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:10:24.071044 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Create backup started, backup: 017a133b-0b87-4753-be0e-99e8ef4fe027 volume: cd3e50ad-c2c4-48eb-9560-8a0740a6943e.
Dec 18 21:10:24.086382 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:10:24.087968 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.104289 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.016s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:24.104698 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[a3fdf9c0-eff3-4f51-b8f8-f81c5f2e1e95]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:24.106726 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.114712 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5d068ca6-d0de-49ec-8dfc-961278297366]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:10:24.115509 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:10:24.116843 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[5d068ca6-d0de-49ec-8dfc-961278297366]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:24.117302 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.126486 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3dd98f10-568d-47fc-a0ff-fb49e45c06c8]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:10:24.127339 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:10:24.128427 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[3dd98f10-568d-47fc-a0ff-fb49e45c06c8]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:24.131412 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.148318 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.016s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:24.148933 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[1ad0f9b4-c29d-49f5-9e30-5ca0aa1cdccc]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:24.151103 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.178956 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.027s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:24.179720 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[75e2538b-5df1-404a-aa9f-ec6459544c94]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:24.182886 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.199637 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.016s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:24.201806 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[22ecf3f1-0d10-4b70-912a-d7422c07da93]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:24.204586 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:24.235975 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] CMD "nvme version" returned: 0 in 0.032s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:24.237387 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] <== get_connector_properties: return (150ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:10:25.348795 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:10:25.350734 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991b145e0>, 'connection_properties': {'name': 'volumes/volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'cd3e50ad-c2c4-48eb-9560-8a0740a6943e', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:10:25.351881 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:10:25.427889 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] <== connect_volume: return (76ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991b3d490>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:10:25.436543 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:10:25.460128 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Starting backup of volume='cd3e50ad-c2c4-48eb-9560-8a0740a6943e'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 21:10:25.461019 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Volume file is RBD: attempting optimized backup {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Dec 18 21:10:25.492858 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Creating base image 'volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' {{(pid=134059) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Dec 18 21:10:25.493695 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 21:10:25.543706 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Using --from-snap 'None' for incremental backup of volume cd3e50ad-c2c4-48eb-9560-8a0740a6943e. {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Dec 18 21:10:25.544125 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Creating backup snapshot='backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732' {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Dec 18 21:10:26.307829 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Performing differential transfer from 'volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e' to 'volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 21:10:26.308160 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_oujhj0i4 --pool volumes volumes/volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e@backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 21:10:26.308539 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 21:10:33.683376 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Differential backup transfer completed in 7.3754s {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Dec 18 21:10:33.683994 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 21:10:33.696087 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 21:10:33.696838 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 21:10:33.705944 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 21:10:33.706360 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 21:10:33.716541 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}}
Dec 18 21:10:33.717492 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Backing up metadata for volume cd3e50ad-c2c4-48eb-9560-8a0740a6943e. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 21:10:33.783459 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Backup '017a133b-0b87-4753-be0e-99e8ef4fe027' of volume cd3e50ad-c2c4-48eb-9560-8a0740a6943e finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 21:10:33.785389 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991b145e0>, 'connection_properties': {'name': 'volumes/volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'cd3e50ad-c2c4-48eb-9560-8a0740a6943e', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991b3d490>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:10:33.812071 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] <== disconnect_volume: return (25ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:10:33.892409 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-ad7315d4-d074-4b41-95b5-5736f112a752 tempest-TestVolumeBackupRestore-2104047399 None] Create backup finished. backup: 017a133b-0b87-4753-be0e-99e8ef4fe027.
Dec 18 21:10:34.111063 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:10:34.111886 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:10:35.735624 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Restore backup started, backup: 017a133b-0b87-4753-be0e-99e8ef4fe027 volume: 4d1a0629-0c83-4586-abdd-6a090066b332.
Dec 18 21:10:35.748786 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:10:35.750299 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.768724 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.018s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:35.769060 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[cf9ada91-069e-4f4f-ae9f-3d73f80b6a22]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:35.770425 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.777818 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[01d3b647-46fd-4566-a8f9-eb1b2db4981d]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:10:35.778566 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:10:35.780239 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[01d3b647-46fd-4566-a8f9-eb1b2db4981d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:35.785599 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.806150 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b56d8726-debf-4980-8d51-e0029a75783f]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[b56d8726-debf-4980-8d51-e0029a75783f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:35.810665 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.820372 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.010s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:35.821265 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[3be9201a-b21a-48b1-98da-66b3f1a6b3d8]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:35.822498 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.841391 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.018s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:35.842066 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[6a04881e-7d20-4dec-befe-18caefcf3b81]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:35.846429 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.855406 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:35.855406 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[d595a090-f7ad-4343-be1e-16b4e1af308c]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:10:35.855406 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:10:35.873605 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:10:35.874789 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] <== get_connector_properties: return (125ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:10:36.660852 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:10:36.662913 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991c9d8e0>, 'connection_properties': {'name': 'volumes/volume-4d1a0629-0c83-4586-abdd-6a090066b332', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4d1a0629-0c83-4586-abdd-6a090066b332', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:10:36.666467 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:10:36.720857 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] <== connect_volume: return (56ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f999124b8e0>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:10:36.728152 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:10:36.739197 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Starting restore from Ceph backup=017a133b-0b87-4753-be0e-99e8ef4fe027 to volume=volume-4d1a0629-0c83-4586-abdd-6a090066b332 {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}}
Dec 18 21:10:36.816799 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Looking for snapshot of backup base 'volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:10:36.817440 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Found snapshot 'backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 21:10:36.822999 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Volume file is RBD. {{(pid=134059) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}}
Dec 18 21:10:36.865471 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Attempting differential restore. {{(pid=134059) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}}
Dec 18 21:10:36.865755 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Attempting incremental restore from base='volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' snap='backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732' {{(pid=134059) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}}
Dec 18 21:10:36.866153 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Performing differential transfer from 'volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' to 'volume-4d1a0629-0c83-4586-abdd-6a090066b332' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 21:10:36.866595 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027@backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 21:10:36.866961 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_jvdkfxa8 --pool volumes - volumes/volume-4d1a0629-0c83-4586-abdd-6a090066b332' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 21:10:41.936766 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Restore transfer completed in 5.0698s {{(pid=134059) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}}
Dec 18 21:10:41.938501 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}}
Dec 18 21:10:41.985814 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Restoring volume base metadata {{(pid=134059) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Dec 18 21:10:42.017013 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] No metadata of type 'volume-metadata' to restore {{(pid=134059) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Dec 18 21:10:42.017394 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Restoring volume glance metadata {{(pid=134059) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}}
Dec 18 21:10:42.229163 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Restore to volume 4d1a0629-0c83-4586-abdd-6a090066b332 finished successfully. {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}}
Dec 18 21:10:42.230758 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f9991c9d8e0>, 'connection_properties': {'name': 'volumes/volume-4d1a0629-0c83-4586-abdd-6a090066b332', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4d1a0629-0c83-4586-abdd-6a090066b332', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f999124b8e0>}, 'force': True, 'ignore_errors': False}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:10:42.243399 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] <== disconnect_volume: return (11ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:10:42.311492 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.manager [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Encryption key ID for volume 4d1a0629-0c83-4586-abdd-6a090066b332 already matches encryption key ID in backup 017a133b-0b87-4753-be0e-99e8ef4fe027. {{(pid=134059) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}}
Dec 18 21:10:42.384476 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-8511de16-14d3-4188-879c-a69f4bc96b8c tempest-TestVolumeBackupRestore-2104047399 None] Finished restoring backup 017a133b-0b87-4753-be0e-99e8ef4fe027 to volume 4d1a0629-0c83-4586-abdd-6a090066b332.
Dec 18 21:11:34.109292 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:11:34.109982 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:11:59.908343 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Delete backup started, backup: 017a133b-0b87-4753-be0e-99e8ef4fe027.
Dec 18 21:11:59.910848 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Delete started for backup=017a133b-0b87-4753-be0e-99e8ef4fe027 {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 21:11:59.912188 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Trying diff format basename='volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' for backup base image of volume cd3e50ad-c2c4-48eb-9560-8a0740a6943e. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 21:12:00.003479 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Looking for snapshot of backup base 'volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:12:00.004182 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Found snapshot 'backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 21:12:00.004621 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Deleting backup snapshot='backup.017a133b-0b87-4753-be0e-99e8ef4fe027.snap.1639861825.543732' {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Dec 18 21:12:00.815013 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Deleting backup base image='volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' of volume cd3e50ad-c2c4-48eb-9560-8a0740a6943e.
Dec 18 21:12:01.747345 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Base backup image='volume-cd3e50ad-c2c4-48eb-9560-8a0740a6943e.backup.017a133b-0b87-4753-be0e-99e8ef4fe027' of volume cd3e50ad-c2c4-48eb-9560-8a0740a6943e deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 21:12:01.814954 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Delete of backup '017a133b-0b87-4753-be0e-99e8ef4fe027' for volume 'cd3e50ad-c2c4-48eb-9560-8a0740a6943e' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 21:12:01.824971 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:12:01.867976 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Created reservations ['0737dd99-0acc-4382-b3f0-ae74405c54da', '750e9ebc-db9a-49df-bfbe-9338df2beb8d'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 21:12:01.935659 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-2bb9be0b-7cf0-467a-810e-873a2198650b tempest-TestVolumeBackupRestore-2104047399 None] Delete backup finished, backup 017a133b-0b87-4753-be0e-99e8ef4fe027 deleted.
Dec 18 21:12:34.106114 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:12:34.106800 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:13:34.102944 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:13:34.103935 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:14:34.100630 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:14:34.101234 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:15:34.097882 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:15:34.099983 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:15:36.393636 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Create backup started, backup: c096624b-0e3f-435a-b6ee-bb6e405458ed volume: 711fd27e-147d-4084-8318-a72e449932b9.
Dec 18 21:15:36.405277 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:15:36.425658 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.428164 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.018s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:36.428705 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[37938586-7ae1-436a-bd81-d45a54570ac8]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:36.433667 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.440655 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f7005097-bd6f-48f9-9b67-0d2487c1e988]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:15:36.442692 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:15:36.446565 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[f7005097-bd6f-48f9-9b67-0d2487c1e988]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:36.452153 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.461858 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e7b548ee-0231-4d81-aafc-9599f6fb28a1]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:15:36.462540 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:15:36.464597 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[e7b548ee-0231-4d81-aafc-9599f6fb28a1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:36.466111 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.496945 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.030s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:36.497363 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[d957d8dc-dd44-41b2-a973-2624505fb633]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:36.498662 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.524895 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.025s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:36.525563 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[9adcca0a-549a-4d5d-8800-f21c15a23d26]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:36.530914 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.540151 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:36.540639 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[b6121c14-ffee-4591-9436-873255c1b76a]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:36.542387 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:36.567761 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:36.569369 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] <== get_connector_properties: return (163ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:15:37.439455 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:15:37.440863 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f99912750d0>, 'connection_properties': {'name': 'volumes/volume-711fd27e-147d-4084-8318-a72e449932b9', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '711fd27e-147d-4084-8318-a72e449932b9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:15:37.442706 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:15:37.507392 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] <== connect_volume: return (66ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f99911c3130>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:15:37.516013 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:15:37.556433 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Starting backup of volume='711fd27e-147d-4084-8318-a72e449932b9'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 21:15:37.557752 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Volume file is RBD: attempting optimized backup {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Dec 18 21:15:37.634076 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Creating base image 'volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' {{(pid=134059) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Dec 18 21:15:37.634598 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 21:15:37.685277 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Using --from-snap 'None' for incremental backup of volume 711fd27e-147d-4084-8318-a72e449932b9. {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Dec 18 21:15:37.686451 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Creating backup snapshot='backup.c096624b-0e3f-435a-b6ee-bb6e405458ed.snap.1639862137.6857908' {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Dec 18 21:15:38.596585 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Performing differential transfer from 'volume-711fd27e-147d-4084-8318-a72e449932b9' to 'volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 21:15:38.597253 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_vqys_yyl --pool volumes volumes/volume-711fd27e-147d-4084-8318-a72e449932b9@backup.c096624b-0e3f-435a-b6ee-bb6e405458ed.snap.1639862137.6857908 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 21:15:38.598090 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 21:15:39.667521 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Differential backup transfer completed in 1.0716s {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Dec 18 21:15:39.669102 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 21:15:39.683350 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 21:15:39.684071 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 21:15:39.694882 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 21:15:39.695680 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 21:15:39.702750 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] No metadata type 'volume-glance-metadata' available {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Dec 18 21:15:39.703647 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Backing up metadata for volume 711fd27e-147d-4084-8318-a72e449932b9. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 21:15:39.803818 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Backup 'c096624b-0e3f-435a-b6ee-bb6e405458ed' of volume 711fd27e-147d-4084-8318-a72e449932b9 finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 21:15:39.805736 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f99912750d0>, 'connection_properties': {'name': 'volumes/volume-711fd27e-147d-4084-8318-a72e449932b9', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '711fd27e-147d-4084-8318-a72e449932b9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f99911c3130>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:15:39.821667 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] <== disconnect_volume: return (14ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:15:39.899223 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-e68add05-a95d-497b-b345-88cde8b4712c tempest-VolumesBackupsTest-1696781328 None] Create backup finished. backup: c096624b-0e3f-435a-b6ee-bb6e405458ed.
Dec 18 21:15:40.673113 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Delete backup started, backup: c096624b-0e3f-435a-b6ee-bb6e405458ed.
Dec 18 21:15:40.677136 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Delete started for backup=c096624b-0e3f-435a-b6ee-bb6e405458ed {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 21:15:40.678096 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Trying diff format basename='volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' for backup base image of volume 711fd27e-147d-4084-8318-a72e449932b9. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 21:15:40.761867 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Looking for snapshot of backup base 'volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:15:40.762282 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Found snapshot 'backup.c096624b-0e3f-435a-b6ee-bb6e405458ed.snap.1639862137.6857908' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 21:15:40.763506 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Deleting backup snapshot='backup.c096624b-0e3f-435a-b6ee-bb6e405458ed.snap.1639862137.6857908' {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Dec 18 21:15:41.702759 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Deleting backup base image='volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' of volume 711fd27e-147d-4084-8318-a72e449932b9.
Dec 18 21:15:42.160016 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Base backup image='volume-711fd27e-147d-4084-8318-a72e449932b9.backup.c096624b-0e3f-435a-b6ee-bb6e405458ed' of volume 711fd27e-147d-4084-8318-a72e449932b9 deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 21:15:42.261546 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Delete of backup 'c096624b-0e3f-435a-b6ee-bb6e405458ed' for volume '711fd27e-147d-4084-8318-a72e449932b9' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 21:15:42.267320 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:15:42.297608 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Created reservations ['246b9242-efc7-4917-b60c-b737202d11d3', '9a690dbd-4ab9-4c20-bf49-68cf99838008'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 21:15:42.346185 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-84b22569-d812-4f47-a2bb-2a15f3754f77 tempest-VolumesBackupsTest-1696781328 None] Delete backup finished, backup c096624b-0e3f-435a-b6ee-bb6e405458ed deleted.
Dec 18 21:15:53.582834 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Create backup started, backup: a36d0633-f6c2-46c3-99db-31abf7e74b6d volume: 115c0e6c-fd0b-4fa6-af13-338519528865.
Dec 18 21:15:53.615204 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:15:53.621642 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.651542 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.033s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:53.655642 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[161a7273-0eca-4f90-b8e3-157667d15f72]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:53.655642 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.676792 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1a7ae573-04e9-4584-b3a5-f294810c580e]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:15:53.677605 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:15:53.680645 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[1a7ae573-04e9-4584-b3a5-f294810c580e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:53.682175 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.707805 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5b300177-ce4a-4154-8d2e-53f4d627e269]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:15:53.708263 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:15:53.709605 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[5b300177-ce4a-4154-8d2e-53f4d627e269]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:53.710909 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.750108 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.039s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:53.750825 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[e827bc42-270d-488f-b0ef-8f0b4cc4c04f]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:53.752578 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.770507 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.017s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:53.771026 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[9692fe1e-41e7-4909-8f10-386d1f681efa]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:53.774929 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.791974 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.010s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:53.792354 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[04bf4c20-4c6d-467d-8ae0-92313ec70a27]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:15:53.793106 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:15:53.818067 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:15:53.819179 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] <== get_connector_properties: return (202ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:15:54.692063 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:15:54.694582 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f999118d4f0>, 'connection_properties': {'name': 'volumes/volume-115c0e6c-fd0b-4fa6-af13-338519528865', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '115c0e6c-fd0b-4fa6-af13-338519528865', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:15:54.695671 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:15:54.821898 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] <== connect_volume: return (126ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f999118d700>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:15:54.831323 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:15:54.851278 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Starting backup of volume='115c0e6c-fd0b-4fa6-af13-338519528865'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 21:15:54.852458 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Volume file is RBD: attempting optimized backup {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}}
Dec 18 21:15:54.885903 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Creating base image 'volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' {{(pid=134059) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}}
Dec 18 21:15:54.886749 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 21:15:54.956500 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Using --from-snap 'None' for incremental backup of volume 115c0e6c-fd0b-4fa6-af13-338519528865. {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}}
Dec 18 21:15:54.956943 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Creating backup snapshot='backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525' {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}}
Dec 18 21:15:55.572330 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Performing differential transfer from 'volume-115c0e6c-fd0b-4fa6-af13-338519528865' to 'volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 21:15:55.572846 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_2ywmii9p --pool volumes volumes/volume-115c0e6c-fd0b-4fa6-af13-338519528865@backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 21:15:55.573300 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 21:16:02.795930 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Differential backup transfer completed in 7.2230s {{(pid=134059) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}}
Dec 18 21:16:02.797646 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 21:16:02.814805 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 21:16:02.816543 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 21:16:02.826364 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 21:16:02.827710 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 21:16:02.838896 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}}
Dec 18 21:16:02.839810 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Backing up metadata for volume 115c0e6c-fd0b-4fa6-af13-338519528865. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 21:16:02.907458 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Backup 'a36d0633-f6c2-46c3-99db-31abf7e74b6d' of volume 115c0e6c-fd0b-4fa6-af13-338519528865 finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 21:16:02.909991 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f999118d4f0>, 'connection_properties': {'name': 'volumes/volume-115c0e6c-fd0b-4fa6-af13-338519528865', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '115c0e6c-fd0b-4fa6-af13-338519528865', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f999118d700>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:16:02.965975 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] <== disconnect_volume: return (54ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:16:03.064399 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-b88cb73f-7564-47f0-a03f-74ccc9b4565a tempest-VolumesBackupsTest-1696781328 None] Create backup finished. backup: a36d0633-f6c2-46c3-99db-31abf7e74b6d.
Dec 18 21:16:05.490512 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Restore backup started, backup: a36d0633-f6c2-46c3-99db-31abf7e74b6d volume: 252e16fa-3cfe-4246-a51b-ba1b118c4103.
Dec 18 21:16:05.510318 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:16:05.520335 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.533807 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.022s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:16:05.535537 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[7e600cf4-9cd9-4935-8a73-5568e97f042c]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:16:05.537882 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.549059 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3de20c4d-08ff-4fdc-adc2-9b2f8386a8df]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:16:05.549648 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:16:05.551232 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[3de20c4d-08ff-4fdc-adc2-9b2f8386a8df]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:16:05.552653 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.569982 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[93be7f87-9a00-409d-84d4-4b94c8e76229]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:16:05.570438 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:16:05.572350 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[93be7f87-9a00-409d-84d4-4b94c8e76229]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:16:05.575661 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.596350 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.020s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:16:05.597450 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[6c86cd45-22e0-47ea-8ca1-0377bd543122]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:16:05.598962 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.616948 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.017s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:16:05.617865 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[82538b4f-7518-4343-b2cf-b93d66d2d842]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:16:05.621143 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.635907 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.016s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:16:05.636531 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[f80e37e7-2266-46cb-bc08-fc7c96f723a8]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:16:05.639424 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:16:05.674827 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] CMD "nvme version" returned: 0 in 0.035s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:16:05.677362 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] <== get_connector_properties: return (166ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:16:06.675463 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:16:06.677062 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f999118da90>, 'connection_properties': {'name': 'volumes/volume-252e16fa-3cfe-4246-a51b-ba1b118c4103', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '252e16fa-3cfe-4246-a51b-ba1b118c4103', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:16:06.678417 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:16:06.767341 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] <== connect_volume: return (88ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991b2cdf0>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:16:06.784014 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:16:06.802093 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Starting restore from Ceph backup=a36d0633-f6c2-46c3-99db-31abf7e74b6d to volume=volume-252e16fa-3cfe-4246-a51b-ba1b118c4103 {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}}
Dec 18 21:16:06.934356 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Looking for snapshot of backup base 'volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:16:06.935904 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Found snapshot 'backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 21:16:06.944739 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Volume file is RBD. {{(pid=134059) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}}
Dec 18 21:16:07.020400 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Attempting differential restore. {{(pid=134059) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}}
Dec 18 21:16:07.021398 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Attempting incremental restore from base='volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' snap='backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525' {{(pid=134059) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}}
Dec 18 21:16:07.021998 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Performing differential transfer from 'volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' to 'volume-252e16fa-3cfe-4246-a51b-ba1b118c4103' {{(pid=134059) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}}
Dec 18 21:16:07.022623 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d@backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525 -' into... {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}}
Dec 18 21:16:07.023160 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_y6ux51dy --pool volumes - volumes/volume-252e16fa-3cfe-4246-a51b-ba1b118c4103' {{(pid=134059) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}}
Dec 18 21:16:12.829438 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Restore transfer completed in 5.8068s {{(pid=134059) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}}
Dec 18 21:16:12.831474 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}}
Dec 18 21:16:12.883560 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Restoring volume base metadata {{(pid=134059) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}}
Dec 18 21:16:12.909909 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] No metadata of type 'volume-metadata' to restore {{(pid=134059) put /opt/stack/cinder/cinder/backup/driver.py:345}}
Dec 18 21:16:12.910402 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Restoring volume glance metadata {{(pid=134059) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}}
Dec 18 21:16:13.093444 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Restore to volume 252e16fa-3cfe-4246-a51b-ba1b118c4103 finished successfully. {{(pid=134059) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}}
Dec 18 21:16:13.095972 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f999118da90>, 'connection_properties': {'name': 'volumes/volume-252e16fa-3cfe-4246-a51b-ba1b118c4103', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '252e16fa-3cfe-4246-a51b-ba1b118c4103', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f9991b2cdf0>}, 'force': True, 'ignore_errors': False}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:16:13.123397 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] <== disconnect_volume: return (27ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:16:13.185840 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.manager [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Encryption key ID for volume 252e16fa-3cfe-4246-a51b-ba1b118c4103 already matches encryption key ID in backup a36d0633-f6c2-46c3-99db-31abf7e74b6d. {{(pid=134059) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}}
Dec 18 21:16:13.260835 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-a1e00301-e328-4a0e-b6a4-55d78c141743 tempest-VolumesBackupsTest-1696781328 None] Finished restoring backup a36d0633-f6c2-46c3-99db-31abf7e74b6d to volume 252e16fa-3cfe-4246-a51b-ba1b118c4103.
Dec 18 21:16:16.407342 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Delete backup started, backup: a36d0633-f6c2-46c3-99db-31abf7e74b6d.
Dec 18 21:16:16.410146 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Delete started for backup=a36d0633-f6c2-46c3-99db-31abf7e74b6d {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 21:16:16.411380 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Trying diff format basename='volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' for backup base image of volume 115c0e6c-fd0b-4fa6-af13-338519528865. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 21:16:16.527905 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Looking for snapshot of backup base 'volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:16:16.528459 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Found snapshot 'backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}}
Dec 18 21:16:16.528987 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Deleting backup snapshot='backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d.snap.1639862154.956525' {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}}
Dec 18 21:16:16.797131 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Deleting backup base image='volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' of volume 115c0e6c-fd0b-4fa6-af13-338519528865.
Dec 18 21:16:17.481204 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Base backup image='volume-115c0e6c-fd0b-4fa6-af13-338519528865.backup.a36d0633-f6c2-46c3-99db-31abf7e74b6d' of volume 115c0e6c-fd0b-4fa6-af13-338519528865 deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 21:16:17.579199 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Delete of backup 'a36d0633-f6c2-46c3-99db-31abf7e74b6d' for volume '115c0e6c-fd0b-4fa6-af13-338519528865' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 21:16:17.588101 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:16:17.639975 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Created reservations ['b2abb569-0f7e-4f07-a532-5502b6c10565', 'b91f2dfb-3df5-4ebe-adcc-74f43780fd09'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 21:16:17.691988 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-aaa2cc7c-fba8-4064-9868-097decfb1461 tempest-VolumesBackupsTest-1696781328 None] Delete backup finished, backup a36d0633-f6c2-46c3-99db-31abf7e74b6d deleted.
Dec 18 21:16:34.098157 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:16:34.098860 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:17:06.381613 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Create backup started, backup: 3f8bc3b3-85c6-4476-aa10-8bece5d06351 volume: 453214a7-66ea-4df2-bb71-f35aa118fee6 snapshot: 2660e5ba-05af-4047-b2fa-0f83ae62129e.
Dec 18 21:17:06.403765 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.13', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:17:06.405372 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.436990 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.031s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:17:06.437538 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[c659af13-1a0f-445a-a669-60633c79ca6b]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator.  The InitiatorName must be unique\n## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ae8da8de7c81\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:17:06.442197 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.461327 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2485813b-0314-4b31-9db7-5509c17c4a71]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:17:06.461666 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:17:06.463098 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[2485813b-0314-4b31-9db7-5509c17c4a71]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:17:06.466577 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.479325 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}}
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e273cfd0-7309-4523-9820-e63bdcb0eed5]: [Errno 2] No such file or directory: 'systool' {{(pid=153537) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}}
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]: Traceback (most recent call last):
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     ret = func(*f_args, **f_kwargs)
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     return func(*args, **kwargs)
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     return putils.execute(on_execute=on_execute,
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     obj = subprocess.Popen(cmd,
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 18 21:17:06.479844 ncdv-master-1 cinder-backup[134059]: FileNotFoundError: [Errno 2] No such file or directory: 'systool'
Dec 18 21:17:06.481225 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[e273cfd0-7309-4523-9820-e63bdcb0eed5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:17:06.487075 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.505289 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.020s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:17:06.505743 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c3b6866-d690-4427-b6e2-0b476175916c]: (4, ('/dev/sda1\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:17:06.510211 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.521441 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.013s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:17:06.522125 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5d7cfca-6421-425c-95f7-3cee9ce51bdb]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:17:06.524213 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.542600 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.018s {{(pid=153537) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:17:06.543216 ncdv-master-1 cinder-backup[134059]: DEBUG oslo.privsep.daemon [-] privsep: reply[c075f7ff-a0f1-4273-879c-a0ec003f5907]: (4, ('238f7184-2d59-3249-a469-f87fee7acbc8\n', '')) {{(pid=153537) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}}
Dec 18 21:17:06.544160 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Running cmd (subprocess): nvme version {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Dec 18 21:17:06.583741 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_concurrency.processutils [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] CMD "nvme version" returned: 0 in 0.039s {{(pid=134059) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Dec 18 21:17:06.585804 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.utils [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] <== get_connector_properties: return (180ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.13', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ae8da8de7c81', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '238f7184-2d59-3249-a469-f87fee7acbc8', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71a52c12-b412-495b-8039-bf02a6407b89'} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:17:08.072008 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connector [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Factory for rbd on None {{(pid=134059) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}}
Dec 18 21:17:08.075208 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f99912757c0>, 'connection_properties': {'name': 'volumes/volume-b2cc5dd1-96f1-46b9-87ee-fa96efb6245f', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b2cc5dd1-96f1-46b9-87ee-fa96efb6245f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:17:08.077873 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.linuxrbd [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] opening connection to ceph cluster (timeout=-1). {{(pid=134059) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}}
Dec 18 21:17:08.203457 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] <== connect_volume: return (127ms) {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f99911c0b50>} {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:17:08.215644 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:17:08.239831 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Starting backup of volume='453214a7-66ea-4df2-bb71-f35aa118fee6'. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}}
Dec 18 21:17:08.306352 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134059) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}}
Dec 18 21:17:08.306897 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Creating backup base image='volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.base' for volume 453214a7-66ea-4df2-bb71-f35aa118fee6. {{(pid=134059) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}}
Dec 18 21:17:08.373255 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Copying data from volume 453214a7-66ea-4df2-bb71-f35aa118fee6. {{(pid=134059) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}}
Dec 18 21:17:08.404675 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferring data between 'volume-453214a7-66ea-4df2-bb71-f35aa118fee6' and 'volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.base' {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}}
Dec 18 21:17:08.405087 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] 8 chunks of 134217728 bytes to be transferred {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}}
Dec 18 21:17:09.993949 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 1 of 8 (82545K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:11.470794 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 2 of 8 (88777K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:12.726148 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 3 of 8 (104554K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:14.122148 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 4 of 8 (93933K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:15.383903 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 5 of 8 (103944K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:16.661154 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 6 of 8 (102737K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:18.066742 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 7 of 8 (93442K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:19.385628 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Transferred chunk 8 of 8 (99482K/s) {{(pid=134059) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}}
Dec 18 21:17:19.433801 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Getting metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}}
Dec 18 21:17:19.445508 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134059) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}}
Dec 18 21:17:19.445926 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Getting metadata type 'volume-metadata' {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}}
Dec 18 21:17:19.452535 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] No metadata type 'volume-metadata' available {{(pid=134059) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}}
Dec 18 21:17:19.453106 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Getting metadata type 'volume-glance-metadata' {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}}
Dec 18 21:17:19.460389 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.driver [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] No metadata type 'volume-glance-metadata' available {{(pid=134059) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}}
Dec 18 21:17:19.461005 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Backing up metadata for volume 453214a7-66ea-4df2-bb71-f35aa118fee6. {{(pid=134059) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}}
Dec 18 21:17:19.524386 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Backup '3f8bc3b3-85c6-4476-aa10-8bece5d06351' of volume 453214a7-66ea-4df2-bb71-f35aa118fee6 finished. {{(pid=134059) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}}
Dec 18 21:17:19.525758 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] ==> disconnect_volume: call "{'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7f99912757c0>, 'connection_properties': {'name': 'volumes/volume-b2cc5dd1-96f1-46b9-87ee-fa96efb6245f', 'hosts': ['10.222.0.13'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b2cc5dd1-96f1-46b9-87ee-fa96efb6245f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7f99911c0b50>}, 'force': True, 'ignore_errors': True}" {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}}
Dec 18 21:17:19.542114 ncdv-master-1 cinder-backup[134059]: DEBUG os_brick.initiator.connectors.rbd [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] <== disconnect_volume: return (15ms) None {{(pid=134059) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}}
Dec 18 21:17:19.632362 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-7cf4f7a1-6990-452e-aa61-6bb9abcf075b tempest-VolumesSnapshotTestJSON-1921935471 None] Create backup finished. backup: 3f8bc3b3-85c6-4476-aa10-8bece5d06351.
Dec 18 21:17:19.792020 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Delete backup started, backup: 3f8bc3b3-85c6-4476-aa10-8bece5d06351.
Dec 18 21:17:19.794522 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Delete started for backup=3f8bc3b3-85c6-4476-aa10-8bece5d06351 {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}}
Dec 18 21:17:19.795233 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Trying diff format basename='volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.3f8bc3b3-85c6-4476-aa10-8bece5d06351' for backup base image of volume 453214a7-66ea-4df2-bb71-f35aa118fee6. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}}
Dec 18 21:17:19.845194 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Image 'volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.3f8bc3b3-85c6-4476-aa10-8bece5d06351' not found - trying diff format name {{(pid=134059) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}}
Dec 18 21:17:19.880258 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Looking for snapshot of backup base 'volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.base' {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}}
Dec 18 21:17:19.880565 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Backup base 'volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.base' has no snapshots {{(pid=134059) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}}
Dec 18 21:17:19.881116 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] No backup snapshot to delete {{(pid=134059) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}}
Dec 18 21:17:19.894504 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Deleting backup base image='volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.base' of volume 453214a7-66ea-4df2-bb71-f35aa118fee6.
Dec 18 21:17:20.907917 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Base backup image='volume-453214a7-66ea-4df2-bb71-f35aa118fee6.backup.base' of volume 453214a7-66ea-4df2-bb71-f35aa118fee6 deleted. {{(pid=134059) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}}
Dec 18 21:17:21.001619 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.backup.drivers.ceph [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Delete of backup '3f8bc3b3-85c6-4476-aa10-8bece5d06351' for volume '453214a7-66ea-4df2-bb71-f35aa118fee6' finished. {{(pid=134059) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}}
Dec 18 21:17:21.009393 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_db.sqlalchemy.engines [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134059) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Dec 18 21:17:21.048120 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.quota [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Created reservations ['21e93224-935a-47cc-87fc-1dad3e57cd1a', '41486f5d-b743-4b72-96d3-10e646e8e214'] {{(pid=134059) reserve /opt/stack/cinder/cinder/quota.py:755}}
Dec 18 21:17:21.088573 ncdv-master-1 cinder-backup[134059]: INFO cinder.backup.manager [None req-4e7e5edb-7789-471a-b691-1beac19d32f3 tempest-VolumesSnapshotTestJSON-1921935471 None] Delete backup finished, backup 3f8bc3b3-85c6-4476-aa10-8bece5d06351 deleted.
Dec 18 21:17:34.099272 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:17:34.100233 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:18:34.101996 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:18:34.102647 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:19:34.107664 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:19:34.108316 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:20:34.112555 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:20:34.113294 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:21:34.109244 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:21:34.109946 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:22:34.114628 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:22:34.115280 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:23:34.120106 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:23:34.120767 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:24:34.125468 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:24:34.126138 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:25:34.130322 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:25:34.131006 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:26:34.134761 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:26:34.135415 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:27:34.139388 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:27:34.140070 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:28:34.144495 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:28:34.145168 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:30:34.141543 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:30:34.142222 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:31:34.147277 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:31:34.147970 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:32:34.152020 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:32:34.152693 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:33:34.157109 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:33:34.157769 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:34:34.161648 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:34:34.162401 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:35:34.167218 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:35:34.167772 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:36:34.172565 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:36:34.173231 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:37:34.177958 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:37:34.178633 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:38:34.175849 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:38:34.176541 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:39:34.181836 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:39:34.182427 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:40:34.186922 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:40:34.187591 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:41:34.191444 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:41:34.192145 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:42:34.197543 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:42:34.198275 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:43:34.193457 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:43:34.194174 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:44:34.196159 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:44:34.196805 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:45:34.200772 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:45:34.201403 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:46:34.205577 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:46:34.206671 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:47:34.202765 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:47:34.203319 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:48:34.207198 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:48:34.207924 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:49:34.212248 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:49:34.212875 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:50:34.211834 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:50:34.212551 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:51:34.217181 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:51:34.217895 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:52:34.221823 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:52:34.222701 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:53:34.226942 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:53:34.227515 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:54:34.231938 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:54:34.232612 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:55:34.237045 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:55:34.237695 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:56:34.241685 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:56:34.242347 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:57:34.246784 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:57:34.247337 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:58:34.252452 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:58:34.253360 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 21:59:34.249873 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 21:59:34.250529 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:00:34.255123 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:00:34.255677 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:01:34.260486 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:01:34.261064 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:02:34.256076 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:02:34.256625 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:03:34.252001 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:03:34.252526 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:04:34.257566 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:04:34.258880 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:05:34.254743 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:05:34.255515 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:06:34.259244 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:06:34.259849 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:07:34.255738 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:07:34.256436 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:08:34.261016 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:08:34.261712 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:09:34.266223 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:09:34.267364 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:10:34.272079 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:10:34.272770 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:11:34.277146 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:11:34.277810 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:12:34.282077 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:12:34.282790 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:13:34.287565 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:13:34.288193 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Dec 18 22:14:34.292452 ncdv-master-1 cinder-backup[134059]: DEBUG oslo_service.periodic_task [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134059) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Dec 18 22:14:34.293123 ncdv-master-1 cinder-backup[134059]: DEBUG cinder.manager [None req-754e89d6-f364-42b6-8d92-878acf55116d None None] Notifying Schedulers of capabilities ... {{(pid=134059) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}