-- Logs begin at Sat 2022-09-24 20:01:15 UTC, end at Sat 2022-09-24 22:17:41 UTC. --
Sep 24 20:40:10.303241 ncdv-master-1 systemd[1]: Started Devstack devstack@c-vol.service.
Sep 24 20:40:13.506815 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "singleton_lock" {{(pid=133251) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}}
Sep 24 20:40:13.507477 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=133251) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:315}}
Sep 24 20:40:13.507477 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=133251) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:333}}
Sep 24 20:40:13.924858 ncdv-master-1 cinder-volume[133251]: INFO dbcounter [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Registered counter for database cinder
Sep 24 20:40:13.945856 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_db.sqlalchemy.engines [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133251) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 24 20:40:13.952669 ncdv-master-1 cinder-volume[133251]: DEBUG dbcounter [-] [133251] Writer thread running {{(pid=133251) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}}
Sep 24 20:40:14.260328 ncdv-master-1 cinder-volume[133251]: INFO cinder.rpc [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version.
Sep 24 20:40:14.265676 ncdv-master-1 cinder-volume[133251]: INFO cinder.rpc [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version.
Sep 24 20:40:14.273193 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_db.sqlalchemy.engines [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133251) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 24 20:40:14.342376 ncdv-master-1 cinder-volume[133251]: INFO cinder.volume.manager [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Determined volume DB was empty at startup.
Sep 24 20:40:14.343068 ncdv-master-1 cinder-volume[133251]: DEBUG cinder.volume.manager [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Cinder Volume DB check: vol_db_empty=True {{(pid=133251) __init__ /opt/stack/cinder/cinder/volume/manager.py:290}}
Sep 24 20:40:14.352611 ncdv-master-1 cinder-volume[133251]: INFO cinder.volume.manager [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Service not found for updating active_backend_id, assuming default for driver init.
Sep 24 20:40:14.353422 ncdv-master-1 cinder-volume[133251]: WARNING stevedore.named [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager
Sep 24 20:40:14.353883 ncdv-master-1 cinder-volume[133251]: WARNING castellan.key_manager [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
Sep 24 20:40:14.355057 ncdv-master-1 cinder-volume[133251]: WARNING cinder.keymgr.conf_key_mgr [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] This key manager is insecure and is not recommended for production deployments: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
Sep 24 20:40:14.373939 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_db.sqlalchemy.engines [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133251) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 24 20:40:14.380770 ncdv-master-1 cinder-volume[133251]: INFO cinder.rpc [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version.
Sep 24 20:40:14.386326 ncdv-master-1 cinder-volume[133251]: INFO cinder.rpc [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version.
Sep 24 20:40:14.387160 ncdv-master-1 cinder-volume[133251]: WARNING stevedore.named [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager
Sep 24 20:40:14.388162 ncdv-master-1 cinder-volume[133251]: WARNING castellan.key_manager [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
Sep 24 20:40:14.390060 ncdv-master-1 cinder-volume[133251]: INFO cinder.volume.manager [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Image-volume cache enabled for host ncdv-master-1@ceph.
Sep 24 20:40:14.397811 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_db.sqlalchemy.engines [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133251) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 24 20:40:14.419937 ncdv-master-1 cinder-volume[133251]: INFO oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Starting 1 workers
Sep 24 20:40:14.425123 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Started child 133423 {{(pid=133251) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Sep 24 20:40:14.426233 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Full set of CONF: {{(pid=133251) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}}
Sep 24 20:40:14.427461 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ******************************************************************************** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}}
Sep 24 20:40:14.427994 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Configuration options gathered from: {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}}
Sep 24 20:40:14.428500 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}}
Sep 24 20:40:14.429233 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}}
Sep 24 20:40:14.429685 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ================================================================================ {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}}
Sep 24 20:40:14.430625 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] allow_availability_zone_fallback = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.431199 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] allow_compression_on_image_upload = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.431759 ncdv-master-1 cinder-volume[133423]: INFO cinder.service [-] Starting cinder-volume node (version 21.1.0)
Sep 24 20:40:14.432458 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] allowed_direct_url_schemes     = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.433010 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] api_rate_limit                 = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.433690 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] auth_strategy                  = keystone {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.434297 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] az_cache_duration              = 3600 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.435367 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_availability_zone      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.435979 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_host                   = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.436559 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_name                   = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.437135 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_stats_polling_interval = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.437682 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backup_api_class               = cinder.backup.api.API {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.438051 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backup_manager                 = cinder.backup.manager.BackupManager {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.438666 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backup_name_template           = backup-%s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.439242 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backup_use_same_host           = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.439683 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backup_use_temp_snapshot       = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.440254 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] chap_password                  = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.440834 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] chap_username                  =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.441399 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] chiscsi_conf                   = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.441976 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_internal_tenant_project_id = 8e368838cf0844c799bbf8bbd5b5a803 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.442556 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_internal_tenant_user_id = c66ad10d7496429da3b81648cf9e83f3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.443105 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cloned_volume_same_az          = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.443646 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cluster                        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.444206 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] compression_format             = gzip {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.444833 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] compute_api_class              = cinder.compute.nova.API {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.445497 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] config_dir                     = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.445970 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] config_file                    = ['/etc/cinder/cinder.conf'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.446523 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] config_source                  = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.447054 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] consistencygroup_api_class     = cinder.consistencygroup.api.API {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.447634 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] control_exchange               = openstack {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.448220 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] debug                          = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.448788 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] default_availability_zone      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.449291 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] default_group_type             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.450690 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.451284 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] default_volume_type            = ceph {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.451888 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] driver_client_cert             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.452430 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] driver_client_cert_key         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.452941 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] driver_data_namespace          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.453457 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] driver_ssl_cert_path           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.453972 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] driver_ssl_cert_verify         = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.454490 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] driver_use_ssl                 = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.455073 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] enable_force_upload            = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.455561 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] enable_new_services            = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.456102 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] enable_unsupported_driver      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.456638 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] enabled_backends               = ['ceph'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.457202 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] enforce_multipath_for_image_xfer = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.457706 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] filter_function                = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.458224 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_api_insecure            = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.458811 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_api_servers             = ['http://10.222.0.52/image'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.459335 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_api_ssl_compression     = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.459703 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_ca_certificates_file    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.460245 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_catalog_info            = image:glance:publicURL {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.460752 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_certfile                = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.461403 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_core_properties         = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.461950 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_keyfile                 = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.462768 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_num_retries             = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.463275 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] glance_request_timeout         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.463800 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] goodness_function              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.465183 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] graceful_shutdown_timeout      = 5 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.465714 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] group_api_class                = cinder.group.api.API {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.466621 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] host                           = ncdv-master-1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.467173 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iet_conf                       = /etc/iet/ietd.conf {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.467747 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_compress_on_upload       = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.468272 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_conversion_address_space_limit = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.468798 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_conversion_cpu_limit     = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.469538 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_conversion_dir           = /opt/stack/data/cinder/conversion {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.470157 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_conversion_disable       = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.470697 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_upload_use_cinder_backend = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.471235 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_upload_use_internal_tenant = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.471349 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Starting volume driver RBDDriver (1.3.0)
Sep 24 20:40:14.471844 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_volume_cache_enabled     = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.472360 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_volume_cache_max_count   = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.472886 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] image_volume_cache_max_size_gb = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.473290 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:14.473618 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] init_host_max_objects_retrieval = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.474128 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] instance_format                = [instance: %(uuid)s]  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.474698 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] instance_uuid_format           = [instance: %(uuid)s]  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.475257 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iscsi_iotype                   = fileio {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.475798 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iscsi_secondary_ip_addresses   = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.476324 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iscsi_target_flags             =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.476844 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iscsi_write_cache              = on {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.477385 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iser_helper                    = tgtadm {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.478789 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iser_ip_address                = 10.222.0.52 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.479414 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iser_port                      = 3260 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.479801 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] iser_target_prefix             = iqn.2010-10.org.openstack: {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.480208 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_config_append              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.480614 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_date_format                = %Y-%m-%d %H:%M:%S {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.480978 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_dir                        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.482471 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_file                       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.482821 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_options                    = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.483227 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_rotate_interval            = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.483644 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_rotate_interval_type       = days {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.484026 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] log_rotation_type              = none {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.484335 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] logging_context_format_string  = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.484644 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] logging_debug_format_suffix    = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.485025 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] logging_default_format_string  = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.485391 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] logging_exception_prefix       = ERROR %(name)s %(instance)s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.485844 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] logging_user_identity_format   = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.486395 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] max_age                        = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.486947 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] max_logfile_count              = 30 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.487352 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] max_logfile_size_mb            = 200 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.487750 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] max_over_subscription_ratio    = 20.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.488143 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] message_reap_interval          = 86400 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.488519 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] message_ttl                    = 2592000 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.488912 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] migration_create_volume_timeout_secs = 300 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.489276 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] monkey_patch                   = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.489670 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] monkey_patch_modules           = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.489982 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] my_ip                          = 10.222.0.52 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.490525 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] no_snapshot_gb_quota           = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.490900 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] num_iser_scan_tries            = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.491303 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] num_shell_tries                = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.491660 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] num_volume_device_scan_tries   = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.492367 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] nvmet_ns_id                    = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.492709 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] nvmet_port_id                  = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.493131 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_max_limit                = 1000 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.493495 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_volume_ext_list          = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.493918 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_volume_extension         = ['cinder.api.contrib.standard_extensions'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.494366 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_volume_listen            = 0.0.0.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.494908 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_volume_listen_port       = 8776 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.495544 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_volume_use_ssl           = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.496400 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] osapi_volume_workers           = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.497499 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] per_volume_size_limit          = -1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.497734 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] periodic_fuzzy_delay           = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.498247 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] periodic_interval              = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.498696 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] project_id_regex               = [0-9a-f\-]+ {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.500671 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] public_endpoint                = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.501432 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] publish_errors                 = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.501953 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_backup_gigabytes         = 1000 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.502614 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_backups                  = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.503242 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_consistencygroups        = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.503812 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_driver                   = cinder.quota.DbQuotaDriver {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.504205 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_gigabytes                = 1000 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.504568 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_groups                   = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.505224 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_snapshots                = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.505627 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] quota_volumes                  = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.505989 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] rate_limit_burst               = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.506397 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] rate_limit_except_level        = CRITICAL {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.506786 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] rate_limit_interval            = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.507153 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] reinit_driver_count            = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.507716 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] replication_device             = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.507872 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] report_discard_supported       = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.508229 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] report_interval                = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.508700 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] reservation_clean_interval     = 86400 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.509000 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] reservation_expire             = 86400 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.509355 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] reserved_percentage            = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.509754 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] resource_query_filters_file    = /etc/cinder/resource_filters.json {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.510118 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] rootwrap_config                = /etc/cinder/rootwrap.conf {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.510563 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] rpc_response_timeout           = 120 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.511094 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] run_external_periodic_tasks    = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.511628 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] scheduler_manager              = cinder.scheduler.manager.SchedulerManager {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.512023 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] scst_target_driver             = iscsi {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.512411 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] scst_target_iqn_name           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.513292 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_down_time              = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.513740 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] snapshot_name_template         = snapshot-%s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.514085 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] snapshot_same_host             = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.514440 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] split_loggers                  = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.514800 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] state_path                     = /opt/stack/data/cinder {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.515195 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] storage_availability_zone      = nova {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.515583 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] storage_protocol               = iSCSI {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.516013 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] syslog_log_facility            = LOG_USER {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.516380 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] target_helper                  = lioadm {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.516871 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] target_ip_address              = 10.222.0.52 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.517258 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] target_port                    = 3260 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.517594 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] target_prefix                  = iqn.2010-10.org.openstack: {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.517983 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] target_protocol                = iscsi {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.518332 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] trace_flags                    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.518752 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] transfer_api_class             = cinder.transfer.api.API {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.519116 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] transport_url                  = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.519515 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] until_refresh                  = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.519867 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_chap_auth                  = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.520218 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_default_quota_class        = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.520568 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_eventlog                   = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.520948 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_journal                    = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.521284 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_json                       = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.521685 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_multipath_for_image_xfer   = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.522008 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_stderr                     = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.522383 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] use_syslog                     = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.522921 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] verify_glance_signatures       = enabled {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.523288 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_api_class               = cinder.volume.api.API {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.523861 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_backend_name            = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.525217 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_clear                   = zero {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.525920 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_clear_ionice            = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.526451 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_clear_size              = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.527016 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_copy_blkio_cgroup_name  = cinder-volume-copy {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.527384 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_copy_bps_limit          = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.527761 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_dd_blocksize            = 1M {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.528144 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_manager                 = cinder.volume.manager.VolumeManager {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.528525 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_name_template           = volume-%s {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.528915 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_service_inithost_offload = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.529567 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volume_usage_audit_period      = month {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.530042 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] volumes_dir                    = /opt/stack/data/cinder/volumes {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.530398 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] watch_log_file                 = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.530773 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] zoning_mode                    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 24 20:40:14.531262 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] os_brick.lock_path             = /opt/stack/data/cinder {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.531709 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.532128 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.532553 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_concurrency.disable_process_locking = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.533363 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_concurrency.lock_path     = /opt/stack/data/cinder {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.533968 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.534341 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.535103 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_metrics.metrics_process_name =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.535487 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.535864 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.536369 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.admin_password = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.536855 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.537375 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.admin_token = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.537911 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.admin_user  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.538457 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_admin_prefix =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.538996 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_host   = 127.0.0.1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.539541 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_port   = 35357 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.540097 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_protocol = https {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.540665 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_section = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.541195 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_type   = password {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.541752 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_uri    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.542277 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.auth_version = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.542809 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.cache       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.543367 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.cafile      = /opt/stack/data/ca-bundle.pem {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.544449 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.certfile    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.545265 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.delay_auth_decision = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.545847 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.546377 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.http_connect_timeout = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.546954 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.547608 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.identity_uri = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.549436 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.include_service_catalog = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.550023 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.insecure    = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.550587 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.interface   = public {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.551460 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.keyfile     = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.552151 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.552678 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.553199 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.553723 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.554266 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.554812 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.555210 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:14.555550 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.556075 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.556658 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.557180 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.region_name = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.557721 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.558237 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.service_token_roles_required = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.558784 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.service_type = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.559332 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.token_cache_time = 300 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.560631 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.561338 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.backend                  = dogpile.cache.null {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.561889 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.backend_argument         = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.562427 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.config_prefix            = cache.oslo {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.563026 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.dead_timeout             = 60.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.563586 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.debug_cache_backend      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.564129 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.enable_retry_client      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.564652 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.enable_socket_keepalive  = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.565172 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.enabled                  = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.565677 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.expiration_time          = 600 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.566195 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.hashclient_retry_attempts = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.566862 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.hashclient_retry_delay   = 1.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.567418 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_dead_retry      = 300 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.567979 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_password        =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.568520 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.569059 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.569580 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_pool_maxsize    = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.570205 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.570759 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_sasl_enabled    = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.571305 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_servers         = ['localhost:11211'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.571823 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_socket_timeout  = 1.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.572347 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.memcache_username        =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.572873 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.proxies                  = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.573387 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.retry_attempts           = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.573895 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.retry_delay              = 0.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.574428 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.socket_keepalive_count   = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.574994 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.socket_keepalive_idle    = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.575510 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.socket_keepalive_interval = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.576010 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.tls_allowed_ciphers      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.576507 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.tls_cafile               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.577027 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.tls_certfile             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.577564 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.tls_enabled              = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.578079 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cache.tls_keyfile              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.578766 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] coordination.backend_url       = etcd3+http://10.222.0.52:2379 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.579372 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.backend               = sqlalchemy {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.579848 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.connection            = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.580413 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.connection_debug      = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.580990 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.connection_parameters =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.581531 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.connection_recycle_time = 3600 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.582074 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.connection_trace      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.583222 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.db_inc_retry_interval = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.583745 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.db_max_retries        = 20 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.584446 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.db_max_retry_interval = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.584993 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.db_retry_interval     = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.585630 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.max_overflow          = 50 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.586139 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.max_pool_size         = 5 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.586726 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.max_retries           = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.587234 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.mysql_enable_ndb      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.587772 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.mysql_sql_mode        = TRADITIONAL {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.588287 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.pool_timeout          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.588894 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.retry_interval        = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.589421 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.slave_connection      = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.589949 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.sqlite_synchronous    = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.590476 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] database.use_db_reconnect      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.591070 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.enforce_new_defaults = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.591604 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.enforce_scope      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.592141 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.policy_default_rule = default {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.592880 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.policy_dirs        = ['policy.d'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.593423 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.policy_file        = policy.yaml {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.594020 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.594551 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.595118 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.595644 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.596183 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.596783 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.connection_string     = messaging:// {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.597237 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.enabled               = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.597765 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.es_doc_type           = notification {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.598269 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.es_scroll_size        = 10000 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.598826 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.es_scroll_time        = 2m {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.599370 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.filter_error_trace    = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.599975 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.hmac_keys             = SECRET_KEY {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.600491 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.sentinel_service_name = mymaster {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.601057 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.socket_timeout        = 0.1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.601611 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] profiler.trace_sqlalchemy      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.603502 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] privsep_osbrick.capabilities   = [21] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.604093 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] privsep_osbrick.group          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.604623 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] privsep_osbrick.helper_command = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.605149 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] privsep_osbrick.logger_name    = oslo_privsep.daemon {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.605948 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.607028 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] privsep_osbrick.user           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.607629 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_sys_admin.capabilities  = [0, 1, 2, 3, 12, 21] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.608138 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_sys_admin.group         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.608632 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_sys_admin.helper_command = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.609146 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_sys_admin.logger_name   = oslo_privsep.daemon {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.609640 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.610379 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] cinder_sys_admin.user          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.611691 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.auth_section      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.612231 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.auth_type         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.612845 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.cafile            = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.613378 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.certfile          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.613922 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.collect_timing    = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.614526 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.insecure          = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.615049 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.keyfile           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.615564 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.send_service_user_token = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.616111 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.split_loggers     = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.616698 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] service_user.timeout           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.617451 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.backend_availability_zone = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.617988 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.backend_native_threads_pool_size = 20 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.618628 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.backup_use_temp_snapshot = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.619175 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.chap_password = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.619714 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.chap_username =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.620244 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.chiscsi_conf  = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.620778 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.621300 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.622071 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.driver_client_cert = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.622636 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.driver_client_cert_key = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.623161 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.driver_data_namespace = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.623675 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.624201 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.624706 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.driver_use_ssl = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.625238 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.enable_deferred_deletion = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.625744 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.enable_unsupported_driver = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.626263 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.626901 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.extra_capabilities = {} {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.627419 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.filter_function = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.627927 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.goodness_function = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.628459 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iet_conf      = /etc/iet/ietd.conf {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.628957 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.629468 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.629974 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.630482 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.631078 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.631598 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iscsi_iotype  = fileio {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.632123 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.632661 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iscsi_target_flags =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.633206 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iscsi_write_cache = on {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.633725 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iser_helper   = tgtadm {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.634414 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iser_ip_address = 10.222.0.52 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.634972 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iser_port     = 3260 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.635739 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.636273 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.636795 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.637303 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.num_shell_tries = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.638418 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.638990 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.nvmet_ns_id   = 10 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.639513 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.nvmet_port_id = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.640032 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.640559 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rados_connection_interval = 5 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.641077 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rados_connection_retries = 3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.641614 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_ceph_conf =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.642402 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.643588 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.644143 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.644663 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.645208 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_pool      = rbd {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.645732 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_secret_uuid = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.646257 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.646961 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.rbd_user      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.647487 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.648015 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.replication_device = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.648538 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.report_discard_supported = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.649051 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.649574 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.reserved_percentage = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.650105 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.scst_target_driver = iscsi {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.650834 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.scst_target_iqn_name = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.651371 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.storage_protocol = iSCSI {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.651887 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.suppress_requests_ssl_warnings = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.652424 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.target_helper = tgtadm {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.653049 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.target_ip_address = 10.222.0.52 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.653617 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.target_port   = 3260 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.654156 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.654971 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.target_protocol = iscsi {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.655472 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.trace_flags   = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.655991 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.unique_fqdn_network = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.656508 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.use_chap_auth = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.657028 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.657533 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_backend_name = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.658083 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_clear  = zero {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.658613 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_clear_ionice = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.659115 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_clear_size = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.659657 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.660192 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.660722 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.661269 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.661898 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] backend_defaults.volumes_dir   = /opt/stack/data/cinder/volumes {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.662449 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_reports.file_event_handler = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.663034 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.663566 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_reports.log_dir           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.664241 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.backend_availability_zone = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.664752 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.backend_host              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.665268 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.backend_native_threads_pool_size = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.665793 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.backup_use_temp_snapshot  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.666305 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.chap_password             = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.666857 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.chap_username             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.667373 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.chiscsi_conf              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.667868 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.deferred_deletion_delay   = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.668370 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.deferred_deletion_purge_interval = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.669436 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.driver_client_cert        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.670075 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.driver_client_cert_key    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.670643 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.driver_data_namespace     = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.671165 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.driver_ssl_cert_path      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.671681 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.driver_ssl_cert_verify    = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.672205 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.driver_use_ssl            = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.672717 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.enable_deferred_deletion  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.673224 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.enable_unsupported_driver = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.673737 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.enforce_multipath_for_image_xfer = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.674260 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.extra_capabilities        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.674810 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.filter_function           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.675308 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.goodness_function         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.675829 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iet_conf                  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.676336 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.image_upload_use_cinder_backend = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.676844 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.image_upload_use_internal_tenant = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.677368 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.image_volume_cache_enabled = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.677883 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.image_volume_cache_max_count = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.678395 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.image_volume_cache_max_size_gb = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.678923 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iscsi_iotype              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.679424 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iscsi_secondary_ip_addresses = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.679943 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iscsi_target_flags        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.680460 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iscsi_write_cache         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.680970 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iser_helper               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.681479 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iser_ip_address           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.681992 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iser_port                 = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.682491 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.iser_target_prefix        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.683038 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.max_over_subscription_ratio = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.683542 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.num_iser_scan_tries       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.684065 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.num_shell_tries           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.684622 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.num_volume_device_scan_tries = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.685118 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.nvmet_ns_id               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.685626 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.nvmet_port_id             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.686297 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rados_connect_timeout     = -1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.687006 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rados_connection_interval = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.687555 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rados_connection_retries  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.688083 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_ceph_conf             = /etc/ceph/ceph.conf {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.688601 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_cluster_name          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.689115 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_exclusive_cinder_pool = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.689658 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_flatten_volume_from_snapshot = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.690237 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_max_clone_depth       = 5 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.690781 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_pool                  = volumes {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.691297 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_secret_uuid           = 9595d6ed-b9df-4104-b762-a42ee9e39a5e {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.691836 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_store_chunk_size      = 4 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.692378 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.rbd_user                  = cinder {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.692887 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.replication_connect_timeout = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.693389 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.replication_device        = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.693899 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.report_discard_supported  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.694427 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.report_dynamic_total_capacity = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.694981 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.reserved_percentage       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.695491 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.scst_target_driver        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.695998 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.scst_target_iqn_name      = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.696504 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.storage_protocol          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.697017 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.suppress_requests_ssl_warnings = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.697533 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.target_helper             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.698035 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.target_ip_address         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.698587 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.target_port               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.699103 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.target_prefix             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.699604 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.target_protocol           = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.700650 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.trace_flags               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.701158 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.unique_fqdn_network       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.701671 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.use_chap_auth             = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.702184 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.use_multipath_for_image_xfer = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.702733 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_backend_name       = ceph {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.703315 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_clear              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.703827 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_clear_ionice       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.704338 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_clear_size         = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.704842 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_copy_blkio_cgroup_name = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.705338 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_copy_bps_limit     = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.705842 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_dd_blocksize       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.706374 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volume_driver             = cinder.volume.drivers.rbd.RBDDriver {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.707182 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ceph.volumes_dir               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.707767 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.708299 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.708826 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.709330 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.709892 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.710436 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.711008 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.711526 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.712042 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.712560 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.713515 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.714040 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.714672 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.715192 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.716450 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.716975 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.717467 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.717973 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.718463 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.719013 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_quroum_max_memory_bytes = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.719527 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_quroum_max_memory_length = 0 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.720052 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.720593 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.721123 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.721653 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.722276 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.ssl      = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.722845 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.ssl_ca_file =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.723384 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.ssl_cert_file =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.723904 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.724436 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.ssl_key_file =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.724977 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_rabbit.ssl_version =  {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.725473 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_notifications.driver = [] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.726032 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_notifications.retry = -1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.726645 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.727179 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] oslo_messaging_notifications.transport_url = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.727695 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] key_manager.backend            = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.728155 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] key_manager.fixed_key          = **** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.728767 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.auth_endpoint         = http://localhost/identity/v3 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.729266 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.barbican_api_version  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.729811 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.barbican_endpoint     = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.730359 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.barbican_endpoint_type = public {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.730980 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.barbican_region_name  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.731511 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.cafile                = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.732006 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.certfile              = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.732663 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.collect_timing        = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.733177 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.insecure              = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.733698 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.keyfile               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.734224 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.number_of_retries     = 60 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.734761 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.retry_delay           = 1 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.735293 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.send_service_user_token = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.735815 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.split_loggers         = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.736327 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.timeout               = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.736830 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.verify_ssl            = True {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.737340 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican.verify_ssl_path       = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.737890 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.auth_section = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.738419 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.auth_type = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.738951 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.cafile   = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.739471 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.certfile = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.739975 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.collect_timing = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.740487 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.insecure = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.740981 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.keyfile  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.741486 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.split_loggers = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.741997 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] barbican_service_user.timeout  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.742555 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.approle_role_id          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.743069 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.approle_secret_id        = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.743568 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.cafile                   = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.744092 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.certfile                 = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.744611 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.collect_timing           = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.745129 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.insecure                 = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.745483 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:14.745802 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.keyfile                  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.746320 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.kv_mountpoint            = secret {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.746924 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.kv_version               = 2 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.747447 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.namespace                = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.747961 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.root_token_id            = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.748479 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.split_loggers            = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.748991 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.ssl_ca_crt_file          = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.749493 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.timeout                  = None {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.750013 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.use_ssl                  = False {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.750581 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] vault.vault_url                = http://127.0.0.1:8200 {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 24 20:40:14.751035 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] ******************************************************************************** {{(pid=133251) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}}
Sep 24 20:40:14.791711 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.driver [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Driver hasn't implemented _init_vendor_properties()
Sep 24 20:40:14.792391 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.driver [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Initialized capabilities list: {'vendor_name': 'Open Source', 'driver_version': '1.3.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:2e6e9886-13df-4b21-824b-dab0a721ddf9:cinder:volumes', 'backend_state': 'up', 'qos_support': True, 'volume_backend_name': 'ceph', 'replication_enabled': False, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=133423) init_capabilities /opt/stack/cinder/cinder/volume/driver.py:812}}
Sep 24 20:40:14.806034 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_db.sqlalchemy.engines [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133423) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 24 20:40:14.808885 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writer thread running {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}}
Sep 24 20:40:14.848535 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.db.sqlalchemy.api [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Building query based on filter {{(pid=133423) _process_snaps_filters /opt/stack/cinder/cinder/db/sqlalchemy/api.py:3885}}
Sep 24 20:40:14.872888 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.image_utils [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Configuration image_conversion_dir is None or the path doesn't exist. {{(pid=133423) cleanup_temporary_file /opt/stack/cinder/cinder/image/image_utils.py:991}}
Sep 24 20:40:14.879392 ncdv-master-1 cinder-volume[133423]: INFO cinder.keymgr.migration [None req-02915ce7-cea3-4ec4-8bb4-a2249a2c2900 None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Sep 24 20:40:14.886829 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:14.914802 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:14.949715 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:40:14.976165 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Driver initialization completed successfully.
Sep 24 20:40:14.977766 ncdv-master-1 cinder-volume[133423]: INFO cinder.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Initiating service 3 cleanup
Sep 24 20:40:14.987249 ncdv-master-1 cinder-volume[133423]: INFO cinder.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Service 3 cleanup completed.
Sep 24 20:40:14.987923 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.service [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Creating RPC server for service cinder-volume {{(pid=133423) start /opt/stack/cinder/cinder/service.py:238}}
Sep 24 20:40:14.998670 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.service [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=133423) start /opt/stack/cinder/cinder/service.py:244}}
Sep 24 20:40:15.091336 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Initializing RPC dependent components of volume driver RBDDriver (1.3.0)
Sep 24 20:40:15.092283 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:15.133948 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:40:15.213599 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-e862dc4b-24dc-4199-96d3-7974635a13e8 None None] Driver post RPC initialization completed successfully.
Sep 24 20:40:24.413271 ncdv-master-1 cinder-volume[133251]: DEBUG dbcounter [-] [133251] Writing DB stats cinder:SELECT=16,cinder:INSERT=1 {{(pid=133251) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:40:25.206790 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=22,cinder:INSERT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:40:25.215398 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:40:45.216176 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:41:05.218857 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:41:05.222722 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:41:25.230817 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:41:35.226169 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:41:38.216896 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:41:38.218143 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:41:38.243503 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:41:38.271959 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:41:45.236176 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:41:55.237027 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:42:15.241529 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:42:25.240014 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:42:38.217597 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:42:38.218270 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:42:38.248053 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:42:38.274445 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:42:45.243890 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:42:45.247035 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:43:05.248755 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:43:25.249509 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:43:35.258733 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:43:38.220535 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:43:38.221206 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:43:38.247390 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:43:38.273606 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:43:45.262699 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:44:05.268494 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:44:15.277988 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:44:25.281509 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:44:38.219346 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:44:38.220460 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:44:38.243370 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:44:38.266533 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:44:45.280906 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:44:55.289501 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:45:15.291626 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:45:25.302209 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:45:38.221179 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:45:38.221842 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:45:38.249459 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:45:38.272415 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:45:45.302793 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:46:05.308719 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:46:15.320281 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:46:35.320481 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:46:35.321180 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:46:38.216969 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:46:38.217622 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:46:38.240898 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:46:38.264161 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:46:45.329422 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:46:55.325618 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:47:05.330203 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:47:25.334099 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:47:25.338801 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:47:38.219166 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:47:38.219835 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:47:38.242325 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:47:38.264142 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:47:45.328980 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:48:05.330674 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:48:05.331270 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:UPDATE=3,cinder:SELECT=7 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:48:25.329464 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:48:35.335701 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:48:38.219345 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:48:38.220556 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:48:38.243394 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:48:38.265559 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:48:45.336621 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=9,cinder:UPDATE=4 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:49:05.343589 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:49:05.344207 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:49:35.354129 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:49:35.355426 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:49:38.220850 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:49:38.222113 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:49:38.283286 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:49:38.347612 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:50:05.364508 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:50:25.367560 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:50:38.223739 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:50:38.224550 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:50:38.275071 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:50:38.341223 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:50:55.369707 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:50:55.370419 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:51:35.378739 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:51:38.227187 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:51:38.230483 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:51:38.293090 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:51:38.398031 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:51:45.384995 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:52:05.386777 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:52:05.388833 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:52:25.394864 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:52:38.223235 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:52:38.224124 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:52:38.292683 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:52:38.363093 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:52:45.402096 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:52:45.404020 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:53:25.401802 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:53:25.402994 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:53:38.228564 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:53:38.229891 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:53:38.259897 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:53:38.283189 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:53:45.404811 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:53:56.288775 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133423) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:53:56.342846 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Flow 'volume_create_manager' (89e149ab-3520-4aed-b136-ee67b434c945) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:53:56.352498 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2a87bd4c-c5b9-4773-8ba4-3f5a84056799) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:53:56.446121 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2a87bd4c-c5b9-4773-8ba4-3f5a84056799) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:53:56Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-LiveMigrationTest-volume-1633407142',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=3c6e1dda-a103-4e0d-b86a-6da231607ef6,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a2b7e30f79bf42e497c53c091ad0a106',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:53:56Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:53:56Z,use_quota=True,user_id='9c03c67e0c4c4de7a186f28ac10d0b92',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:53:56.459689 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9a4c179c-1dee-445c-bacc-45eb13b326b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:53:56.463612 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (9a4c179c-1dee-445c-bacc-45eb13b326b6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:53:56.485199 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4398e454-b2d1-4853-a57c-65ebc94625d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:53:56.493787 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (4398e454-b2d1-4853-a57c-65ebc94625d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'volume_name': 'volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'volume_size': 1}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:53:56.505758 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c2661591-76d6-4f33-b855-cd8c9fd91998) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:53:56.513172 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c2661591-76d6-4f33-b855-cd8c9fd91998) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:53:56.526987 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (97757420-e986-4c2a-9245-4d13c10adc26) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:53:56.530718 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Volume 3c6e1dda-a103-4e0d-b86a-6da231607ef6: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'volume_size': 1}
Sep 24 20:53:56.535053 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] creating volume 'volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6' {{(pid=133423) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1130}}
Sep 24 20:53:56.536212 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:53:56.680029 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:982}}
Sep 24 20:53:56.680489 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:991}}
Sep 24 20:53:56.686112 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (97757420-e986-4c2a-9245-4d13c10adc26) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'volume_size': 1}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:53:56.693431 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (aff72431-a7a3-4727-82f9-5b48e153a87b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:53:56.711595 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Volume volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6 (3c6e1dda-a103-4e0d-b86a-6da231607ef6): created successfully
Sep 24 20:53:56.715417 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (aff72431-a7a3-4727-82f9-5b48e153a87b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:53:56.723038 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Flow 'volume_create_manager' (89e149ab-3520-4aed-b136-ee67b434c945) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:53:56.735069 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-454f6257-42e4-4a30-9cb6-26808237d937 tempest-LiveMigrationTest-471200620 None] Created volume successfully.
Sep 24 20:53:58.896422 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-e5ee1811-dffd-4705-9171-0f7ac04b6eae req-4f0c770e-b66d-46cc-bc17-494683bd3f1c tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:54:00.187039 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-e5ee1811-dffd-4705-9171-0f7ac04b6eae req-4f0c770e-b66d-46cc-bc17-494683bd3f1c tempest-LiveMigrationTest-471200620 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.295s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:54:00.189976 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [req-e5ee1811-dffd-4705-9171-0f7ac04b6eae req-4f0c770e-b66d-46cc-bc17-494683bd3f1c tempest-LiveMigrationTest-471200620 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '9595d6ed-b9df-4104-b762-a42ee9e39a5e', 'volume_id': '3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'discard': True}} {{(pid=133423) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1783}}
Sep 24 20:54:00.301365 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-e5ee1811-dffd-4705-9171-0f7ac04b6eae req-4f0c770e-b66d-46cc-bc17-494683bd3f1c tempest-LiveMigrationTest-471200620 None] Connection info returned from driver {'name': 'volumes/volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '923ca728-1538-4a9c-84c7-6f19dd634598'} {{(pid=133423) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 24 20:54:00.394584 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-e5ee1811-dffd-4705-9171-0f7ac04b6eae req-4f0c770e-b66d-46cc-bc17-494683bd3f1c tempest-LiveMigrationTest-471200620 None] attachment_update completed successfully.
Sep 24 20:54:10.395750 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=17,cinder:UPDATE=2,cinder:DELETE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:54:15.420985 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=24,cinder:UPDATE=7 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:54:19.624492 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-8d5d1ed5-3828-4c21-bd5f-59653dd316f6 tempest-LiveMigrationTest-730417024 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:54:20.805727 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-8d5d1ed5-3828-4c21-bd5f-59653dd316f6 tempest-LiveMigrationTest-730417024 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.181s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:54:20.808146 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-8d5d1ed5-3828-4c21-bd5f-59653dd316f6 tempest-LiveMigrationTest-730417024 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '9595d6ed-b9df-4104-b762-a42ee9e39a5e', 'volume_id': '3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'discard': True}} {{(pid=133423) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1783}}
Sep 24 20:54:20.848080 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-8d5d1ed5-3828-4c21-bd5f-59653dd316f6 tempest-LiveMigrationTest-730417024 None] Connection info returned from driver {'name': 'volumes/volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '3c6e1dda-a103-4e0d-b86a-6da231607ef6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '90464aa9-2c14-4ce8-b396-63da0fe73bd2'} {{(pid=133423) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 24 20:54:21.025713 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-8d5d1ed5-3828-4c21-bd5f-59653dd316f6 tempest-LiveMigrationTest-730417024 None] attachment_update completed successfully.
Sep 24 20:54:38.228210 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:54:38.229717 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:54:38.336043 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:54:38.414079 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:54:45.407055 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:UPDATE=4,cinder:SELECT=15 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:54:46.403329 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=18,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:54:46.411598 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-74a6fabc-9d6d-4e06-9955-a788a02f377a tempest-LiveMigrationTest-730417024 None] Terminate volume connection completed successfully.
Sep 24 20:54:46.412348 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-caf4656a-0c26-41f3-bb1b-7c74149f30b7 req-74a6fabc-9d6d-4e06-9955-a788a02f377a tempest-LiveMigrationTest-730417024 None] Deleting attachment 923ca728-1538-4a9c-84c7-6f19dd634598. {{(pid=133423) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 24 20:54:56.584039 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-4edc67bb-6fe7-4999-a2a9-347200c26dae req-0318c9eb-143a-443c-a99c-010b9154cab7 tempest-LiveMigrationTest-471200620 None] Terminate volume connection completed successfully.
Sep 24 20:54:56.584708 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-4edc67bb-6fe7-4999-a2a9-347200c26dae req-0318c9eb-143a-443c-a99c-010b9154cab7 tempest-LiveMigrationTest-471200620 None] Deleting attachment 90464aa9-2c14-4ce8-b396-63da0fe73bd2. {{(pid=133423) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 24 20:54:58.503654 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133423) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:54:58.526106 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Flow 'volume_create_manager' (24793fd7-c9a3-46bb-870a-60adef45916b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:54:58.532489 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (75ed67cd-2777-4173-ad8d-5aab0b51a3b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:54:58.549886 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (75ed67cd-2777-4173-ad8d-5aab0b51a3b8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:54:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-276314323',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=b1d57646-064a-4739-a13c-b954f6f5cb19,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a2b7e30f79bf42e497c53c091ad0a106',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:54:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:54:58Z,use_quota=True,user_id='9c03c67e0c4c4de7a186f28ac10d0b92',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:54:58.558843 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ee6fcd74-bcc1-400e-b75b-02b4eda8bc34) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:54:58.562861 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ee6fcd74-bcc1-400e-b75b-02b4eda8bc34) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:54:58.568385 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f4d96adb-2e89-4413-9201-41a808d5c467) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:54:58.670779 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (f4d96adb-2e89-4413-9201-41a808d5c467) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'volume_name': 'volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628ad5ac10>}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:54:58.676319 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (630bf5d4-67de-4fd7-ac35-dc8e2b2adde0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:54:58.679990 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (630bf5d4-67de-4fd7-ac35-dc8e2b2adde0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:54:58.684492 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a162415b-f643-4ee0-bd7d-e405d34867a0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:54:58.685558 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume b1d57646-064a-4739-a13c-b954f6f5cb19: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628ad5ac10>}
Sep 24 20:54:58.686285 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Cloning b1d57646-064a-4739-a13c-b954f6f5cb19 from image 4c30e8e9-8fb2-4377-83f1-98322cb17799  at location (None, None). {{(pid=133423) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}}
Sep 24 20:54:58.687111 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.coordination [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Acquiring lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" by "_prepare_image_cache_entry" {{(pid=133423) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:54:58.698876 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.coordination [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" acquired by "_prepare_image_cache_entry" :: waited 0.011s {{(pid=133423) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:54:58.708054 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Preparing cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=133423) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:891}}
Sep 24 20:54:58.709161 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Attempting to retrieve cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=133423) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}}
Sep 24 20:55:15.414997 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=14,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:55:17.209599 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Image fetch details: dest /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph, size 616.00 MB, duration 18.50 sec {{(pid=133423) fetch /opt/stack/cinder/cinder/image/image_utils.py:587}}
Sep 24 20:55:17.210938 ncdv-master-1 cinder-volume[133423]: INFO cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Image download 616.00 MB at 33.31 MB/s
Sep 24 20:55:17.275171 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:17.443763 ncdv-master-1 sudo[145792]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph
Sep 24 20:55:17.444755 ncdv-master-1 sudo[145792]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 24 20:55:19.012879 ncdv-master-1 sudo[145792]: pam_unix(sudo:session): session closed for user root
Sep 24 20:55:19.015062 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph" returned: 0 in 1.753s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:19.015977 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Temporary image 4c30e8e9-8fb2-4377-83f1-98322cb17799 is fetched for user 9c03c67e0c4c4de7a186f28ac10d0b92. {{(pid=133423) fetch /opt/stack/cinder/cinder/image/image_utils.py:1124}}
Sep 24 20:55:19.080309 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:19.177095 ncdv-master-1 sudo[145799]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph
Sep 24 20:55:19.177949 ncdv-master-1 sudo[145799]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 24 20:55:20.544363 ncdv-master-1 sudo[145799]: pam_unix(sudo:session): session closed for user root
Sep 24 20:55:20.547529 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph" returned: 0 in 1.467s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:20.549000 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] creating volume 'volume-b1d57646-064a-4739-a13c-b954f6f5cb19' {{(pid=133423) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1130}}
Sep 24 20:55:20.551297 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:55:20.691607 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:982}}
Sep 24 20:55:20.692002 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:991}}
Sep 24 20:55:20.714253 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.volume_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Attempting download of 4c30e8e9-8fb2-4377-83f1-98322cb17799 ((None, None)) to volume b1d57646-064a-4739-a13c-b954f6f5cb19. {{(pid=133423) copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1190}}
Sep 24 20:55:20.785897 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_download_4c30e8e9-8fb2-4377-83f1-98322cb17799_2umodxr3 {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:20.914329 ncdv-master-1 sudo[145820]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_download_4c30e8e9-8fb2-4377-83f1-98322cb17799_2umodxr3
Sep 24 20:55:20.915164 ncdv-master-1 sudo[145820]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 24 20:55:22.873868 ncdv-master-1 sudo[145820]: pam_unix(sudo:session): session closed for user root
Sep 24 20:55:22.876105 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_download_4c30e8e9-8fb2-4377-83f1-98322cb17799_2umodxr3" returned: 0 in 2.092s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:22.877516 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:22.994378 ncdv-master-1 sudo[145836]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph
Sep 24 20:55:22.995235 ncdv-master-1 sudo[145836]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 24 20:55:24.369974 ncdv-master-1 sudo[145836]: pam_unix(sudo:session): session closed for user root
Sep 24 20:55:24.372875 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph" returned: 0 in 1.495s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:24.373866 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] 4c30e8e9-8fb2-4377-83f1-98322cb17799 was vhdx, converting to raw {{(pid=133423) fetch_to_volume_format /opt/stack/cinder/cinder/image/image_utils.py:825}}
Sep 24 20:55:24.376414 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -f vhdx /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph /opt/stack/data/cinder/conversion/tmpiynfk2hn {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:24.406189 ncdv-master-1 sudo[145842]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -f vhdx /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph /opt/stack/data/cinder/conversion/tmpiynfk2hn
Sep 24 20:55:24.407742 ncdv-master-1 sudo[145842]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 24 20:55:26.906320 ncdv-master-1 sudo[145842]: pam_unix(sudo:session): session closed for user root
Sep 24 20:55:26.912017 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -f vhdx /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph /opt/stack/data/cinder/conversion/tmpiynfk2hn" returned: 0 in 2.532s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:26.912017 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:27.004510 ncdv-master-1 sudo[145857]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph
Sep 24 20:55:27.005408 ncdv-master-1 sudo[145857]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 24 20:55:28.691337 ncdv-master-1 sudo[145857]: pam_unix(sudo:session): session closed for user root
Sep 24 20:55:28.693696 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph" returned: 0 in 1.784s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:28.695567 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Image conversion details: src /opt/stack/data/cinder/conversion/image_fetch_4c30e8e9-8fb2-4377-83f1-98322cb17799_zdfyilulncdv-master-1@ceph, size 700.00 MB, duration 2.53 sec, destination /opt/stack/data/cinder/conversion/tmpiynfk2hn {{(pid=133423) _convert_image /opt/stack/cinder/cinder/image/image_utils.py:419}}
Sep 24 20:55:28.695953 ncdv-master-1 cinder-volume[133423]: INFO cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Converted 700.00 MB image at 276.21 MB/s
Sep 24 20:55:28.697423 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:55:28.804333 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] volume has no backup snaps {{(pid=133423) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1263}}
Sep 24 20:55:28.810997 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume volume-b1d57646-064a-4739-a13c-b954f6f5cb19 is not a clone. {{(pid=133423) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1291}}
Sep 24 20:55:28.830982 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] deleting rbd volume volume-b1d57646-064a-4739-a13c-b954f6f5cb19 {{(pid=133423) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1415}}
Sep 24 20:55:29.169269 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): rbd import --pool volumes --order 22 /opt/stack/data/cinder/conversion/tmpiynfk2hn volume-b1d57646-064a-4739-a13c-b954f6f5cb19 --new-format --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:34.869869 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] CMD "rbd import --pool volumes --order 22 /opt/stack/data/cinder/conversion/tmpiynfk2hn volume-b1d57646-064a-4739-a13c-b954f6f5cb19 --new-format --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 5.700s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:35.022878 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:55:35.232482 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:982}}
Sep 24 20:55:35.233081 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:991}}
Sep 24 20:55:35.233660 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.volume_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Downloaded image 4c30e8e9-8fb2-4377-83f1-98322cb17799 ((None, None)) to volume b1d57646-064a-4739-a13c-b954f6f5cb19 successfully. {{(pid=133423) copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1232}}
Sep 24 20:55:35.402111 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.image_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Temporary image 4c30e8e9-8fb2-4377-83f1-98322cb17799 for user 9c03c67e0c4c4de7a186f28ac10d0b92 is deleted. {{(pid=133423) fetch /opt/stack/cinder/cinder/image/image_utils.py:1128}}
Sep 24 20:55:35.430617 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:55:35.431496 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=16,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:55:35.548028 ncdv-master-1 cinder-volume[133423]: WARNING cinder.quota [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Deprecated: Default quota for resource: volumes_ceph is set by the default quota flag: quota_volumes_ceph, it is now deprecated. Please use the default quota class for default quota.
Sep 24 20:55:35.553158 ncdv-master-1 cinder-volume[133423]: WARNING cinder.quota [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Deprecated: Default quota for resource: gigabytes_ceph is set by the default quota flag: quota_gigabytes_ceph, it is now deprecated. Please use the default quota class for default quota.
Sep 24 20:55:35.688199 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.quota [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Created reservations ['9e39e9ca-9731-43a5-a413-fe29057b5de1', '347e5468-8c53-4e07-a50e-02c8025a021e', 'add7730c-7570-4eae-bd1a-45bc5c63288d', '5b31ed13-185f-4cb7-9260-1acf4c00893a'] {{(pid=133423) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 24 20:55:35.723520 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Creating image volume entry: {'provider_location': None, 'user_id': '9c03c67e0c4c4de7a186f28ac10d0b92', 'replication_status': None, 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'deleted': False, 'host': 'ncdv-master-1@ceph#ceph', 'volume_metadata': [], 'previous_status': None, 'replication_driver_data': None, 'multiattach': False, 'updated_at': datetime.datetime(2022, 9, 24, 20, 54, 58, tzinfo=datetime.timezone.utc), 'consistencygroup_id': None, 'bootable': False, 'provider_geometry': None, 'scheduled_at': datetime.datetime(2022, 9, 24, 20, 54, 58, tzinfo=datetime.timezone.utc), 'availability_zone': 'nova', 'shared_targets': True, 'service_uuid': None, 'project_id': '8e368838cf0844c799bbf8bbd5b5a803', 'volume_glance_metadata': [{'key': 'signature_verified', 'value': 'False'}], 'display_description': None, 'size': 1, 'terminated_at': None, 'volume_admin_metadata': [], 'deleted_at': None, 'encryption_key_id': None, 'ec2_id': None, 'display_name': 'image-4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 54, 58, tzinfo=datetime.timezone.utc), 'replication_extended_status': None, 'provider_id': None, 'group_id': None, 'launched_at': None, 'provider_auth': None, 'cluster_name': None, 'metadata': {}, 'attach_status': 'detached', 'status': 'creating', 'source_volid': 'b1d57646-064a-4739-a13c-b954f6f5cb19'}. {{(pid=133423) _clone_image_volume /opt/stack/cinder/cinder/volume/manager.py:1660}}
Sep 24 20:55:35.836745 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume reschedule parameters: False retry: None {{(pid=133423) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:55:35.863542 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Flow 'volume_create_manager' (3fb58a57-9fd3-4eb0-be9c-c191c67456bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:55:35.870201 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e2f9a805-8a88-43eb-924d-83bd75fde1cc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:35.901812 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e2f9a805-8a88-43eb-924d-83bd75fde1cc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:54:58Z,deleted=False,deleted_at=None,display_description=None,display_name='image-4c30e8e9-8fb2-4377-83f1-98322cb17799',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=a410930c-3d96-4d0b-b7f5-ee675704f604,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='8e368838cf0844c799bbf8bbd5b5a803',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:54:58Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=b1d57646-064a-4739-a13c-b954f6f5cb19,status='creating',terminated_at=None,updated_at=2022-09-24T20:54:58Z,use_quota=True,user_id='9c03c67e0c4c4de7a186f28ac10d0b92',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:35.907359 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (42112966-c27b-427c-99cf-77bd440d5bea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:35.912394 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (42112966-c27b-427c-99cf-77bd440d5bea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:35.917639 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6ea17590-d79a-4d36-a1dc-bf079e749a2b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:35.934675 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6ea17590-d79a-4d36-a1dc-bf079e749a2b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'source_vol', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'volume_name': 'volume-a410930c-3d96-4d0b-b7f5-ee675704f604', 'volume_size': 1, 'source_volid': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'source_volstatus': 'downloading'}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:35.939859 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1bdfdfea-486c-4607-812d-c4ca46f4f1d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:35.944564 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1bdfdfea-486c-4607-812d-c4ca46f4f1d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:35.949311 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1806fc3a-5197-4e3d-a962-e6bee13d3056) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:35.950395 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume a410930c-3d96-4d0b-b7f5-ee675704f604: being created as source_vol with specification: {'status': 'creating', 'volume_name': 'volume-a410930c-3d96-4d0b-b7f5-ee675704f604', 'volume_size': 1, 'source_volid': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'source_volstatus': 'downloading'}
Sep 24 20:55:35.964430 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:55:36.061592 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] creating snapshot='volume-a410930c-3d96-4d0b-b7f5-ee675704f604.clone_snap' {{(pid=133423) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:846}}
Sep 24 20:55:37.003273 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] cloning 'volume-b1d57646-064a-4739-a13c-b954f6f5cb19@volume-a410930c-3d96-4d0b-b7f5-ee675704f604.clone_snap' to 'volume-a410930c-3d96-4d0b-b7f5-ee675704f604' {{(pid=133423) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:852}}
Sep 24 20:55:37.136207 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume volume-b1d57646-064a-4739-a13c-b954f6f5cb19 is not a clone. {{(pid=133423) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1291}}
Sep 24 20:55:37.139538 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:982}}
Sep 24 20:55:37.142403 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:991}}
Sep 24 20:55:37.171184 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] clone created successfully {{(pid=133423) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:925}}
Sep 24 20:55:37.176485 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1806fc3a-5197-4e3d-a962-e6bee13d3056) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-a410930c-3d96-4d0b-b7f5-ee675704f604', 'volume_size': 1, 'source_volid': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'source_volstatus': 'downloading'}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:37.192684 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (a4ef2912-388d-4091-85cd-72ccd1b57f88) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:37.217288 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume volume-a410930c-3d96-4d0b-b7f5-ee675704f604 (a410930c-3d96-4d0b-b7f5-ee675704f604): created successfully
Sep 24 20:55:37.228257 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (a4ef2912-388d-4091-85cd-72ccd1b57f88) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:37.241821 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Flow 'volume_create_manager' (3fb58a57-9fd3-4eb0-be9c-c191c67456bc) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:55:37.262786 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Created volume successfully.
Sep 24 20:55:37.353679 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.cache [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Creating new image-volume cache entry for image 4c30e8e9-8fb2-4377-83f1-98322cb17799 on ncdv-master-1@ceph#ceph {{(pid=133423) create_cache_entry /opt/stack/cinder/cinder/image/cache.py:104}}
Sep 24 20:55:37.365843 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.cache [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] New image-volume cache entry created: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 55, 37, 357882)}. {{(pid=133423) create_cache_entry /opt/stack/cinder/cinder/image/cache.py:129}}
Sep 24 20:55:37.382893 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.coordination [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" released by "_prepare_image_cache_entry" :: held 38.676s {{(pid=133423) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:55:37.382893 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.volume_utils [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Marking volume b1d57646-064a-4739-a13c-b954f6f5cb19 as bootable. {{(pid=133423) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}}
Sep 24 20:55:37.402550 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Copying metadata from image 4c30e8e9-8fb2-4377-83f1-98322cb17799 to b1d57646-064a-4739-a13c-b954f6f5cb19. {{(pid=133423) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:452}}
Sep 24 20:55:37.403451 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Creating volume glance metadata for volume b1d57646-064a-4739-a13c-b954f6f5cb19 backed by image 4c30e8e9-8fb2-4377-83f1-98322cb17799 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': '', 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_name': 'ubuntu-16', 'checksum': '318f990048c8a194829e52141534610d', 'container_format': 'bare', 'disk_format': 'vhdx', 'min_disk': 0, 'min_ram': 0, 'size': 645922816}. {{(pid=133423) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:683}}
Sep 24 20:55:37.659784 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (a162415b-f643-4ee0-bd7d-e405d34867a0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628ad5ac10>}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:37.701752 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (27ea73cf-95e5-426d-83da-124a528812a2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:55:37.729890 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Volume volume-b1d57646-064a-4739-a13c-b954f6f5cb19 (b1d57646-064a-4739-a13c-b954f6f5cb19): created successfully
Sep 24 20:55:37.749563 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (27ea73cf-95e5-426d-83da-124a528812a2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:55:37.765361 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Flow 'volume_create_manager' (24793fd7-c9a3-46bb-870a-60adef45916b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:55:37.797611 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-24bee4ad-a01e-43c3-af20-2f2ecf8c64e4 tempest-LiveMigrationTest-471200620 None] Created volume successfully.
Sep 24 20:55:38.229637 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_service.periodic_task [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133423) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:55:38.232225 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:55:38.316347 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:55:38.388404 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.manager [None req-67b66ea7-3a0c-4316-84b6-51fd64637d6d None None] Notifying Schedulers of capabilities ... {{(pid=133423) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:55:43.720969 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-e93b9309-72e2-4094-9d9b-5eb2a0333f12 req-ce474400-9472-413b-90e5-931933683a6a tempest-LiveMigrationTest-471200620 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:55:45.376763 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-e93b9309-72e2-4094-9d9b-5eb2a0333f12 req-ce474400-9472-413b-90e5-931933683a6a tempest-LiveMigrationTest-471200620 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.656s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:55:45.378823 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [req-e93b9309-72e2-4094-9d9b-5eb2a0333f12 req-ce474400-9472-413b-90e5-931933683a6a tempest-LiveMigrationTest-471200620 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '9595d6ed-b9df-4104-b762-a42ee9e39a5e', 'volume_id': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'discard': True}} {{(pid=133423) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1783}}
Sep 24 20:55:45.459894 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-e93b9309-72e2-4094-9d9b-5eb2a0333f12 req-ce474400-9472-413b-90e5-931933683a6a tempest-LiveMigrationTest-471200620 None] Connection info returned from driver {'name': 'volumes/volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'b99a9535-efd5-43bd-a12a-3b1aaa9d17f1'} {{(pid=133423) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 24 20:55:45.565979 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-e93b9309-72e2-4094-9d9b-5eb2a0333f12 req-ce474400-9472-413b-90e5-931933683a6a tempest-LiveMigrationTest-471200620 None] attachment_update completed successfully.
Sep 24 20:56:05.439639 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=46,cinder:INSERT=12,cinder:UPDATE=17 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:56:15.084388 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:UPDATE=9,cinder:SELECT=55,cinder:INSERT=12,cinder:DELETE=2 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:56:15.103374 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133423) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:56:15.144399 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Flow 'volume_create_manager' (bc10a100-365d-4437-b318-12584d2886ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:56:15.153402 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d3c5118-1edc-4a5e-a739-212135d888e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:15.176679 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3d3c5118-1edc-4a5e-a739-212135d888e1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:56:15Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-LiveAutoBlockMigrationV225Test-volume-2113594978',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='f3c583f2016e4edd94aeec7dd64f9673',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:56:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:56:15Z,use_quota=True,user_id='ca78cf6d34e84f2cb5cb254c5d8af1da',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:15.183161 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e00204f-ef55-4320-8ea6-50f26e0ad2ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:15.187523 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (8e00204f-ef55-4320-8ea6-50f26e0ad2ec) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:15.193713 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6c8ad1cc-98d1-443e-8878-42bd1c7373dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:15.198943 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6c8ad1cc-98d1-443e-8878-42bd1c7373dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': 'f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'volume_name': 'volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'volume_size': 1}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:15.205031 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ecdda390-1dcb-47d7-9e47-e86c9a2639dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:15.209855 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (ecdda390-1dcb-47d7-9e47-e86c9a2639dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:15.215095 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c8e5c33-bc86-4e57-835e-d30f5ed4e387) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:15.216498 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Volume f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'volume_size': 1}
Sep 24 20:56:15.217172 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] creating volume 'volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2' {{(pid=133423) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1130}}
Sep 24 20:56:15.218702 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:15.341715 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:982}}
Sep 24 20:56:15.342984 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133423) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:991}}
Sep 24 20:56:15.347930 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1c8e5c33-bc86-4e57-835e-d30f5ed4e387) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'volume_size': 1}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:15.353158 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (f9fc27e2-d56a-488b-9bd8-090beac5e5d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:15.369695 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Volume volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2 (f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2): created successfully
Sep 24 20:56:15.379071 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (f9fc27e2-d56a-488b-9bd8-090beac5e5d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:15.386518 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Flow 'volume_create_manager' (bc10a100-365d-4437-b318-12584d2886ea) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:56:15.400301 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [None req-1f452070-e268-4ff9-a5e9-6786d26f3914 tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Created volume successfully.
Sep 24 20:56:17.827884 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-dee57e90-006a-4ff6-95f4-a3bf18349f4e req-c616396a-3e11-4f31-91b1-76f78160fd5e tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:56:19.406343 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-dee57e90-006a-4ff6-95f4-a3bf18349f4e req-c616396a-3e11-4f31-91b1-76f78160fd5e tempest-LiveAutoBlockMigrationV225Test-1717881295 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.579s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:56:19.408299 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [req-dee57e90-006a-4ff6-95f4-a3bf18349f4e req-c616396a-3e11-4f31-91b1-76f78160fd5e tempest-LiveAutoBlockMigrationV225Test-1717881295 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '9595d6ed-b9df-4104-b762-a42ee9e39a5e', 'volume_id': 'f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'discard': True}} {{(pid=133423) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1783}}
Sep 24 20:56:19.458376 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-dee57e90-006a-4ff6-95f4-a3bf18349f4e req-c616396a-3e11-4f31-91b1-76f78160fd5e tempest-LiveAutoBlockMigrationV225Test-1717881295 None] Connection info returned from driver {'name': 'volumes/volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'b9160bb8-44bc-4581-9e42-20e1cdc137e6'} {{(pid=133423) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 24 20:56:19.598460 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-dee57e90-006a-4ff6-95f4-a3bf18349f4e req-c616396a-3e11-4f31-91b1-76f78160fd5e tempest-LiveAutoBlockMigrationV225Test-1717881295 None] attachment_update completed successfully.
Sep 24 20:56:21.405704 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-7bf3602c-b477-4aa7-bb0d-f8cfde53b851 tempest-LiveMigrationTest-730417024 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:56:22.807392 ncdv-master-1 cinder-volume[133423]: DEBUG oslo_concurrency.processutils [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-7bf3602c-b477-4aa7-bb0d-f8cfde53b851 tempest-LiveMigrationTest-730417024 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.401s {{(pid=133423) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:56:22.809358 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-7bf3602c-b477-4aa7-bb0d-f8cfde53b851 tempest-LiveMigrationTest-730417024 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '9595d6ed-b9df-4104-b762-a42ee9e39a5e', 'volume_id': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'discard': True}} {{(pid=133423) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1783}}
Sep 24 20:56:22.845357 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-7bf3602c-b477-4aa7-bb0d-f8cfde53b851 tempest-LiveMigrationTest-730417024 None] Connection info returned from driver {'name': 'volumes/volume-b1d57646-064a-4739-a13c-b954f6f5cb19', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'b1d57646-064a-4739-a13c-b954f6f5cb19', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'c86bff85-7cfa-4171-ae67-3cfed70d1254'} {{(pid=133423) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 24 20:56:22.970464 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-7bf3602c-b477-4aa7-bb0d-f8cfde53b851 tempest-LiveMigrationTest-730417024 None] attachment_update completed successfully.
Sep 24 20:56:32.939567 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=24,cinder:DELETE=1,cinder:UPDATE=3 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:56:33.186103 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.manager [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-8d412797-6e75-4f04-9c50-fb9b957d3acf tempest-LiveMigrationTest-730417024 None] Terminate volume connection completed successfully.
Sep 24 20:56:33.189085 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [req-050e2bb8-c461-4738-8320-adc45ddb0f5d req-8d412797-6e75-4f04-9c50-fb9b957d3acf tempest-LiveMigrationTest-730417024 None] Deleting attachment b99a9535-efd5-43bd-a12a-3b1aaa9d17f1. {{(pid=133423) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 24 20:56:36.881432 ncdv-master-1 cinder-volume[133423]: DEBUG dbcounter [-] [133423] Writing DB stats cinder:SELECT=37,cinder:UPDATE=9 {{(pid=133423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:56:36.891790 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133423) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:56:36.939212 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Flow 'volume_create_manager' (06beccda-7b6f-4414-9d35-a406fd3c8ba2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:56:36.946274 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (afd7667a-bae8-4d9d-88b5-b0d9626d3da9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:36.967060 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (afd7667a-bae8-4d9d-88b5-b0d9626d3da9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:56:37Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-267130114',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=12570bdc-0c4e-4a84-9d49-0038138c72bb,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='cfa59dfe189849238f3693696fd1089e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:56:37Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:56:37Z,use_quota=True,user_id='76b517a69e1f4dd0895104befebc128d',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:36.974544 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ef8c085e-06af-48fd-82f7-dc982a91b741) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:36.979503 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (ef8c085e-06af-48fd-82f7-dc982a91b741) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:36.986079 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b4fab09f-7424-4631-8e58-39dad3c5e667) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:37.147783 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b4fab09f-7424-4631-8e58-39dad3c5e667) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '12570bdc-0c4e-4a84-9d49-0038138c72bb', 'volume_name': 'volume-12570bdc-0c4e-4a84-9d49-0038138c72bb', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628b65f850>}' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:37.174372 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (314ebf98-74e8-437a-b13e-cdfefda377b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:37.188192 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (314ebf98-74e8-437a-b13e-cdfefda377b1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:56:37.221124 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.manager [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (f1c1570c-a951-4fa4-a4c5-f9038cb5786e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133423) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:56:37.227813 ncdv-master-1 cinder-volume[133423]: INFO cinder.volume.flows.manager.create_volume [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Volume 12570bdc-0c4e-4a84-9d49-0038138c72bb: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-12570bdc-0c4e-4a84-9d49-0038138c72bb', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628b65f850>}
Sep 24 20:56:37.231750 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Cloning 12570bdc-0c4e-4a84-9d49-0038138c72bb from image 4c30e8e9-8fb2-4377-83f1-98322cb17799  at location (None, None). {{(pid=133423) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}}
Sep 24 20:56:37.233881 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.coordination [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Acquiring lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" by "_prepare_image_cache_entry" {{(pid=133423) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:56:37.257644 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.coordination [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" acquired by "_prepare_image_cache_entry" :: waited 0.025s {{(pid=133423) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:56:37.280177 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.cache [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 56, 37, 265916)}. {{(pid=133423) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:56:37.281477 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.cache [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=133423) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:56:37.282767 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Found cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=133423) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}}
Sep 24 20:56:37.293514 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.coordination [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" released by "_prepare_image_cache_entry" :: held 0.036s {{(pid=133423) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:56:37.295756 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Attempting to retrieve cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=133423) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}}
Sep 24 20:56:37.307881 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.cache [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 56, 37, 303609)}. {{(pid=133423) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:56:37.308611 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.image.cache [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=133423) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:56:37.309445 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.flows.manager.create_volume [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] Creating from source image-volume a410930c-3d96-4d0b-b7f5-ee675704f604 {{(pid=133423) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}}
Sep 24 20:56:37.325555 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133423) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:37.420624 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] creating snapshot='volume-12570bdc-0c4e-4a84-9d49-0038138c72bb.clone_snap' {{(pid=133423) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:846}}
Sep 24 20:56:37.933196 ncdv-master-1 cinder-volume[133423]: DEBUG cinder.volume.drivers.rbd [None req-18d4e270-1693-4c5e-a4d9-7875dbb4b50d tempest-ServersTestBootFromVolume-1605240856 None] cloning 'volume-a410930c-3d96-4d0b-b7f5-ee675704f604@volume-12570bdc-0c4e-4a84-9d49-0038138c72bb.clone_snap' to 'volume-12570bdc-0c4e-4a84-9d49-0038138c72bb' {{(pid=133423) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:852}}
Sep 24 20:56:38.142592 ncdv-master-1 cinder-volume[133423]: python3.8: ../nptl/pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
Sep 24 20:56:38.158291 ncdv-master-1 cinder-volume[133251]: INFO oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Child 133423 killed by signal 6
Sep 24 20:56:38.165021 ncdv-master-1 cinder-volume[133251]: DEBUG oslo_service.service [None req-a735213e-0978-430b-999c-89b6b3401c70 None None] Started child 148026 {{(pid=133251) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Sep 24 20:56:38.183790 ncdv-master-1 cinder-volume[148026]: INFO cinder.service [-] Starting cinder-volume node (version 21.1.0)
Sep 24 20:56:38.282438 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Starting volume driver RBDDriver (1.3.0)
Sep 24 20:56:38.285519 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:38.383906 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:38.517196 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:38.571725 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.driver [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Driver hasn't implemented _init_vendor_properties()
Sep 24 20:56:38.572338 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.driver [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Initialized capabilities list: {'vendor_name': 'Open Source', 'driver_version': '1.3.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:2e6e9886-13df-4b21-824b-dab0a721ddf9:cinder:volumes', 'backend_state': 'up', 'qos_support': True, 'volume_backend_name': 'ceph', 'replication_enabled': False, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=148026) init_capabilities /opt/stack/cinder/cinder/volume/driver.py:812}}
Sep 24 20:56:38.589328 ncdv-master-1 cinder-volume[148026]: DEBUG oslo_db.sqlalchemy.engines [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=148026) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 24 20:56:38.592853 ncdv-master-1 cinder-volume[148026]: DEBUG dbcounter [-] [148026] Writer thread running {{(pid=148026) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}}
Sep 24 20:56:38.727692 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.db.sqlalchemy.api [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Building query based on filter {{(pid=148026) _process_snaps_filters /opt/stack/cinder/cinder/db/sqlalchemy/api.py:3885}}
Sep 24 20:56:38.776048 ncdv-master-1 cinder-volume[148026]: INFO cinder.keymgr.migration [None req-467dcaed-3b3c-42a0-8df1-a69383629116 None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Sep 24 20:56:38.785011 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:38.837577 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:38.901710 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Notifying Schedulers of capabilities ... {{(pid=148026) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:56:38.942730 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Driver initialization completed successfully.
Sep 24 20:56:38.943688 ncdv-master-1 cinder-volume[148026]: INFO cinder.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Initiating service 3 cleanup
Sep 24 20:56:39.057915 ncdv-master-1 cinder-volume[148026]: INFO cinder.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Cleaning Volume with id 12570bdc-0c4e-4a84-9d49-0038138c72bb and status creating
Sep 24 20:56:39.078220 ncdv-master-1 cinder-volume[148026]: INFO cinder.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Service 3 cleanup completed.
Sep 24 20:56:39.078867 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.service [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Creating RPC server for service cinder-volume {{(pid=148026) start /opt/stack/cinder/cinder/service.py:238}}
Sep 24 20:56:39.089212 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.service [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=148026) start /opt/stack/cinder/cinder/service.py:244}}
Sep 24 20:56:39.147181 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Initializing RPC dependent components of volume driver RBDDriver (1.3.0)
Sep 24 20:56:39.148740 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:39.220964 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:39.298416 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [None req-418cf939-4e53-48d9-b89b-342d69ff214c None None] Driver post RPC initialization completed successfully.
Sep 24 20:56:40.176349 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] Acquiring lock "cinder-12570bdc-0c4e-4a84-9d49-0038138c72bb-delete_volume" by "delete_volume" {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:56:40.194200 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] Lock "cinder-12570bdc-0c4e-4a84-9d49-0038138c72bb-delete_volume" acquired by "delete_volume" :: waited 0.018s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:56:40.344258 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:56:40.508661 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] volume has no backup snaps {{(pid=148026) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1263}}
Sep 24 20:56:40.525058 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] deleting rbd volume volume-12570bdc-0c4e-4a84-9d49-0038138c72bb {{(pid=148026) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1415}}
Sep 24 20:56:40.753348 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] volume is a clone so cleaning references {{(pid=148026) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1436}}
Sep 24 20:56:40.787665 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] deleting parent snapshot volume-12570bdc-0c4e-4a84-9d49-0038138c72bb.clone_snap {{(pid=148026) _delete_clone_parent_refs /opt/stack/cinder/cinder/volume/drivers/rbd.py:1329}}
Sep 24 20:57:07.468343 ncdv-master-1 cinder-volume[148026]: DEBUG dbcounter [-] [148026] Writing DB stats cinder:SELECT=15,cinder:UPDATE=1 {{(pid=148026) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:57:07.478286 ncdv-master-1 cinder-volume[148026]: DEBUG dbcounter [-] [148026] Writing DB stats cinder:SELECT=14,cinder:UPDATE=2,cinder:DELETE=1 {{(pid=148026) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:57:07.706578 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] Lock "cinder-12570bdc-0c4e-4a84-9d49-0038138c72bb-delete_volume" released by "delete_volume" :: held 27.511s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server [None req-b1fa3116-43dd-420c-91d8-7a1564581cd0 tempest-ServersTestBootFromVolume-1605240856 None] Exception during message handling: rbd.ReadOnlyImage: [errno 30] RBD read-only image (error removing snapshot b'volume-12570bdc-0c4e-4a84-9d49-0038138c72bb.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     skip_clean = func(self, context, volume, *args, **kwargs)
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return f(*a, **k)
Sep 24 20:57:07.748196 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1042, in delete_volume
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     self._clear_db(volume, new_status)
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise self.value
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1026, in delete_volume
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     self.driver.delete_volume(volume)
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 1438, in delete_volume
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     self._delete_clone_parent_refs(client, parent, parent_snap)
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 1331, in _delete_clone_parent_refs
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     parent_rbd.remove_snap(parent_snap)
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3551, in rbd.Image.remove_snap
Sep 24 20:57:07.749503 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error removing snapshot b'volume-12570bdc-0c4e-4a84-9d49-0038138c72bb.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:07.750990 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:07.801986 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-e64fdef9-1eb3-442b-bc52-c0731ff417d7 tempest-LiveMigrationTest-471200620 None] Terminate volume connection completed successfully.
Sep 24 20:57:07.814141 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-e64fdef9-1eb3-442b-bc52-c0731ff417d7 tempest-LiveMigrationTest-471200620 None] Deleting attachment c86bff85-7cfa-4171-ae67-3cfed70d1254. {{(pid=148026) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 24 20:57:07.847012 ncdv-master-1 cinder-volume[148026]: DEBUG oslo_concurrency.processutils [req-dd28b781-a226-4f88-8606-c599d63bd824 req-ddeefc0f-c5d8-4148-9ae8-51996d8aa1aa tempest-LiveAutoBlockMigrationV225Test-1380422539 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=148026) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 24 20:57:09.516686 ncdv-master-1 cinder-volume[148026]: DEBUG oslo_concurrency.processutils [req-dd28b781-a226-4f88-8606-c599d63bd824 req-ddeefc0f-c5d8-4148-9ae8-51996d8aa1aa tempest-LiveAutoBlockMigrationV225Test-1380422539 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.687s {{(pid=148026) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 24 20:57:09.521446 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-dd28b781-a226-4f88-8606-c599d63bd824 req-ddeefc0f-c5d8-4148-9ae8-51996d8aa1aa tempest-LiveAutoBlockMigrationV225Test-1380422539 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '9595d6ed-b9df-4104-b762-a42ee9e39a5e', 'volume_id': 'f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'discard': True}} {{(pid=148026) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1783}}
Sep 24 20:57:09.624922 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [req-dd28b781-a226-4f88-8606-c599d63bd824 req-ddeefc0f-c5d8-4148-9ae8-51996d8aa1aa tempest-LiveAutoBlockMigrationV225Test-1380422539 None] Connection info returned from driver {'name': 'volumes/volume-f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'f910cbdd-fbe4-4cac-bdb7-3b663c4f26d2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'be10cf03-27e7-40d3-990c-aa09735f2458'} {{(pid=148026) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 24 20:57:09.715036 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Acquiring lock "cinder-b1d57646-064a-4739-a13c-b954f6f5cb19-delete_volume" by "delete_volume" {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:57:09.758467 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Lock "cinder-b1d57646-064a-4739-a13c-b954f6f5cb19-delete_volume" acquired by "delete_volume" :: waited 0.044s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:57:09.777086 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [req-dd28b781-a226-4f88-8606-c599d63bd824 req-ddeefc0f-c5d8-4148-9ae8-51996d8aa1aa tempest-LiveAutoBlockMigrationV225Test-1380422539 None] attachment_update completed successfully.
Sep 24 20:57:09.866703 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:10.029794 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] volume has no backup snaps {{(pid=148026) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1263}}
Sep 24 20:57:10.030264 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] volume has clone snapshot(s) {{(pid=148026) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1378}}
Sep 24 20:57:10.031583 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Volume volume-b1d57646-064a-4739-a13c-b954f6f5cb19 is not a clone. {{(pid=148026) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1291}}
Sep 24 20:57:10.050042 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=148026) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:57:10.164374 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Flow 'volume_create_manager' (ab29efa6-68d1-4638-8eb3-0cbb9e4ed311) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:57:10.176851 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0251bdb9-4db7-4676-8f27-f75d52778965) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:10.234748 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0251bdb9-4db7-4676-8f27-f75d52778965) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:10Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:10.255813 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15685ab7-4874-48c8-a6a9-41b766a67b10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:10.268998 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15685ab7-4874-48c8-a6a9-41b766a67b10) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:10.279524 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16703bfc-2349-4891-9072-12df647634d6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:10.368392 ncdv-master-1 cinder-volume[148026]: WARNING cinder.quota [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Deprecated: Default quota for resource: volumes_ceph is set by the default quota flag: quota_volumes_ceph, it is now deprecated. Please use the default quota class for default quota.
Sep 24 20:57:10.369234 ncdv-master-1 cinder-volume[148026]: WARNING cinder.quota [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Deprecated: Default quota for resource: gigabytes_ceph is set by the default quota flag: quota_gigabytes_ceph, it is now deprecated. Please use the default quota class for default quota.
Sep 24 20:57:10.397597 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.quota [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Created reservations ['50330d1d-dc89-4db6-a352-fb2db613391e', 'f4145d52-7f57-4586-844d-c8ec9fb61d4c', '277b5938-e6c3-4d9e-9388-e90caf87b5f7', '8af4795d-0916-4c2a-9549-25999844d558'] {{(pid=148026) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 24 20:57:10.571998 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16703bfc-2349-4891-9072-12df647634d6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628aa21430>}' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:10.579251 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9a3aa68-193c-4a71-9f6e-e7b68cae9434) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:10.585862 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9a3aa68-193c-4a71-9f6e-e7b68cae9434) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:10.592272 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (16858e71-b19d-4a88-9c7c-4ff673a4e316) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:10.595054 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628aa21430>}
Sep 24 20:57:10.596876 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Cloning f054c8e1-9900-4dfe-8743-857b892350ae from image 4c30e8e9-8fb2-4377-83f1-98322cb17799  at location (None, None). {{(pid=148026) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}}
Sep 24 20:57:10.598792 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Acquiring lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" by "_prepare_image_cache_entry" {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:57:10.603846 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:10.667961 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:10.754969 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.manager [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Notifying Schedulers of capabilities ... {{(pid=148026) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:57:10.761006 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Deleted volume successfully.
Sep 24 20:57:10.773567 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" acquired by "_prepare_image_cache_entry" :: waited 0.175s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:57:10.816451 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [req-34b6d407-41e8-44b4-8d7f-3b7be2cbfcce req-1c9fe1be-b35e-4385-ba38-9866a5bd9cd3 tempest-LiveMigrationTest-471200620 None] Lock "cinder-b1d57646-064a-4739-a13c-b954f6f5cb19-delete_volume" released by "delete_volume" :: held 1.057s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:57:10.818781 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 57, 10, 801810)}. {{(pid=148026) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:57:10.819715 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=148026) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:57:10.820429 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=148026) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}}
Sep 24 20:57:10.833354 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" released by "_prepare_image_cache_entry" :: held 0.060s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:57:10.834867 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Attempting to retrieve cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=148026) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}}
Sep 24 20:57:10.848336 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 57, 10, 840665)}. {{(pid=148026) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:57:10.852573 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=148026) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:57:10.853584 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Creating from source image-volume a410930c-3d96-4d0b-b7f5-ee675704f604 {{(pid=148026) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}}
Sep 24 20:57:10.872484 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:10.974269 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] creating snapshot='volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' {{(pid=148026) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:846}}
Sep 24 20:57:11.040949 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (16858e71-b19d-4a88-9c7c-4ff673a4e316) transitioned into state 'FAILURE' from state 'RUNNING'
Sep 24 20:57:11.040949 ncdv-master-1 cinder-volume[148026]: 5 predecessors (most recent first):
Sep 24 20:57:11.040949 ncdv-master-1 cinder-volume[148026]:   Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f6294f5ef10>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:10Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}, 'provides': None}
Sep 24 20:57:11.041309 ncdv-master-1 cinder-volume[148026]:   |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f6294f5ef10>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:10Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8), 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=4c30e8e9-8fb2-4377-83f1-98322cb17799,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(f054c8e1-9900-4dfe-8743-857b892350ae),volume_id=f054c8e1-9900-4dfe-8743-857b892350ae,volume_properties=VolumeProperties,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628aa21430>}}
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]:      |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume': {'id': 'f054c8e1-9900-4dfe-8743-857b892350ae', '_name_id': None, 'ec2_id': None, 'user_id': '56064ff9075243879d7bce37e3e547e7', 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'name': 'ceph', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-24T20:40:15.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2022-09-24T20:57:09.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'name_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'name': 'ceph', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-24T20:40:15.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'qos_specs': None, 'reservations': ['339bbf31-d5a1-48d8-83d6-ad840029e9a7', '0402076a-a418-41f6-a379-caf6d4d54aff', '9b0ddaf1-365a-4b76-8805-d629cff4e26a', '1c92821c-b0c3-441a-afc4-9e8553564953'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'qos_specs': None, 'reservations': ['339bbf31-d5a1-48d8-83d6-ad840029e9a7', '0402076a-a418-41f6-a379-caf6d4d54aff', '9b0ddaf1-365a-4b76-8805-d629cff4e26a', '1c92821c-b0c3-441a-afc4-9e8553564953'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2022-09-24T20:40:15Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=291311aa-f438-45f1-8d44-6f2e6ad780f8,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2022-09-24T20:40:15Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=291311aa-f438-45f1-8d44-6f2e6ad780f8,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=4c30e8e9-8fb2-4377-83f1-98322cb17799,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(f054c8e1-9900-4dfe-8743-857b892350ae),volume_id=f054c8e1-9900-4dfe-8743-857b892350ae,volume_properties=VolumeProperties,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8)), 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:10Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8), 'context': <cinder.context.RequestContext object at 0x7f6294f5ef10>}, 'provides': None}
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]:         |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f6294f5ef10>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:10Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:10Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:10Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]:            |__Flow 'volume_create_manager': rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 849, in create_cloned_volume
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     src_volume.create_snap(clone_snap)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     result = task.execute(**arguments)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_image(context,
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return r(f, *args, **kwargs)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return fut.result()
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return self.__get_result()
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     raise self._exception
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     result = fn(*args, **kwargs)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_image_cache_or_download(
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update, cloned = self._create_from_image_cache(
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_source_volume(
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 860, in create_cloned_volume
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     src_volume.unprotect_snap(clone_snap)
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:11.041476 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:11.071502 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (16858e71-b19d-4a88-9c7c-4ff673a4e316) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.078616 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (16858e71-b19d-4a88-9c7c-4ff673a4e316) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:11.087824 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9a3aa68-193c-4a71-9f6e-e7b68cae9434) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.093956 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9a3aa68-193c-4a71-9f6e-e7b68cae9434) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:11.102644 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16703bfc-2349-4891-9072-12df647634d6) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.109030 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (16703bfc-2349-4891-9072-12df647634d6) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:11.115984 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15685ab7-4874-48c8-a6a9-41b766a67b10) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.119223 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Updating volume f054c8e1-9900-4dfe-8743-857b892350ae with {'scheduled_at': datetime.datetime(2022, 9, 24, 20, 57, 11, 118803), 'host': None}. {{(pid=148026) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}}
Sep 24 20:57:11.138109 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: re-scheduling SchedulerAPI.create_volume attempt 1 due to [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap') {{(pid=148026) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}}
Sep 24 20:57:11.188655 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: re-scheduled {{(pid=148026) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}}
Sep 24 20:57:11.189745 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:11.246784 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] volume volume-f054c8e1-9900-4dfe-8743-857b892350ae no longer exists in backend
Sep 24 20:57:11.257792 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (15685ab7-4874-48c8-a6a9-41b766a67b10) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True'
Sep 24 20:57:11.267808 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0251bdb9-4db7-4676-8f27-f75d52778965) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.279799 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0251bdb9-4db7-4676-8f27-f75d52778965) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:11.292452 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Flow 'volume_create_manager' (ab29efa6-68d1-4638-8eb3-0cbb9e4ed311) transitioned into state 'REVERTED' from state 'RUNNING'
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Exception during message handling: rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 849, in create_cloned_volume
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     src_volume.create_snap(clone_snap)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     _run_flow()
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     flow_engine.run()
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     failures[0].reraise()
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 350, in reraise
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise value
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image(context,
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return fut.result()
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self.__get_result()
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise self._exception
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image_cache_or_download(
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update, cloned = self._create_from_image_cache(
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_source_volume(
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 860, in create_cloned_volume
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     src_volume.unprotect_snap(clone_snap)
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:11.307673 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:11.417616 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']} {{(pid=148026) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:57:11.496018 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Flow 'volume_create_manager' (0e3214e7-a70c-485f-8692-812ebb5fbdcd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:57:11.516997 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1f7b945d-dc10-4f61-9f71-b5a45a52b0bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.589502 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1f7b945d-dc10-4f61-9f71-b5a45a52b0bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:11Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:11.613679 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c27fcecc-3349-4a4a-9cb7-893b585748ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.634035 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c27fcecc-3349-4a4a-9cb7-893b585748ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:11.649973 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c9b4d246-0961-4de9-a240-88d776f710dd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.730149 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Acquiring lock "cinder-3c6e1dda-a103-4e0d-b86a-6da231607ef6-delete_volume" by "delete_volume" {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:57:11.768794 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Lock "cinder-3c6e1dda-a103-4e0d-b86a-6da231607ef6-delete_volume" acquired by "delete_volume" :: waited 0.038s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:57:11.877906 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c9b4d246-0961-4de9-a240-88d776f710dd) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628abaf670>}' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:11.891579 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96953e14-d853-4bfb-a845-315c921f2c41) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.899537 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96953e14-d853-4bfb-a845-315c921f2c41) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:11.913499 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (77b13636-98ba-49b7-8b2b-18e2a69da3d4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:11.914959 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628abaf670>}
Sep 24 20:57:11.915577 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Cloning f054c8e1-9900-4dfe-8743-857b892350ae from image 4c30e8e9-8fb2-4377-83f1-98322cb17799  at location (None, None). {{(pid=148026) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}}
Sep 24 20:57:11.916103 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Acquiring lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" by "_prepare_image_cache_entry" {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:57:11.946713 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:12.057281 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] volume has no backup snaps {{(pid=148026) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1263}}
Sep 24 20:57:12.057461 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Volume volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6 is not a clone. {{(pid=148026) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1291}}
Sep 24 20:57:12.084362 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] deleting rbd volume volume-3c6e1dda-a103-4e0d-b86a-6da231607ef6 {{(pid=148026) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1415}}
Sep 24 20:57:12.095752 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" acquired by "_prepare_image_cache_entry" :: waited 0.177s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:57:12.132579 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 57, 12, 116791)}. {{(pid=148026) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:57:12.133343 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=148026) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:57:12.138285 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=148026) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}}
Sep 24 20:57:12.148547 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" released by "_prepare_image_cache_entry" :: held 0.053s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:57:12.149727 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Attempting to retrieve cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=148026) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}}
Sep 24 20:57:12.160988 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 57, 12, 156436)}. {{(pid=148026) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:57:12.161975 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=148026) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:57:12.162647 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Creating from source image-volume a410930c-3d96-4d0b-b7f5-ee675704f604 {{(pid=148026) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}}
Sep 24 20:57:12.214947 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:12.411656 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] creating snapshot='volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' {{(pid=148026) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:846}}
Sep 24 20:57:12.454087 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (77b13636-98ba-49b7-8b2b-18e2a69da3d4) transitioned into state 'FAILURE' from state 'RUNNING'
Sep 24 20:57:12.454087 ncdv-master-1 cinder-volume[148026]: 5 predecessors (most recent first):
Sep 24 20:57:12.454087 ncdv-master-1 cinder-volume[148026]:   Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f628ab7bd60>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:11Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}, 'provides': None}
Sep 24 20:57:12.454087 ncdv-master-1 cinder-volume[148026]:   |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f628ab7bd60>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:11Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8), 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=4c30e8e9-8fb2-4377-83f1-98322cb17799,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(f054c8e1-9900-4dfe-8743-857b892350ae),volume_id=f054c8e1-9900-4dfe-8743-857b892350ae,volume_properties=VolumeProperties,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628abaf670>}}
Sep 24 20:57:12.454087 ncdv-master-1 cinder-volume[148026]:      |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume': {'id': 'f054c8e1-9900-4dfe-8743-857b892350ae', '_name_id': None, 'ec2_id': None, 'user_id': '56064ff9075243879d7bce37e3e547e7', 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'name': 'ceph', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-24T20:40:15.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2022-09-24T20:57:09.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'name_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'name': 'ceph', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-24T20:40:15.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'qos_specs': None, 'reservations': ['339bbf31-d5a1-48d8-83d6-ad840029e9a7', '0402076a-a418-41f6-a379-caf6d4d54aff', '9b0ddaf1-365a-4b76-8805-d629cff4e26a', '1c92821c-b0c3-441a-afc4-9e8553564953'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'qos_specs': None, 'reservations': ['339bbf31-d5a1-48d8-83d6-ad840029e9a7', '0402076a-a418-41f6-a379-caf6d4d54aff', '9b0ddaf1-365a-4b76-8805-d629cff4e26a', '1c92821c-b0c3-441a-afc4-9e8553564953'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2022-09-24T20:40:15Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=291311aa-f438-45f1-8d44-6f2e6ad780f8,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2022-09-24T20:40:15Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=291311aa-f438-45f1-8d44-6f2e6ad780f8,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']}}, 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=4c30e8e9-8fb2-4377-83f1-98322cb17799,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(f054c8e1-9900-4dfe-8743-857b892350ae),volume_id=f054c8e1-9900-4dfe-8743-857b892350ae,volume_properties=VolumeProperties,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8)), 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:11Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8), 'context': <cinder.context.RequestContext object at 0x7f628ab7bd60>}, 'provides': None}
Sep 24 20:57:12.454087 ncdv-master-1 cinder-volume[148026]:         |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f628ab7bd60>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:11Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:11Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:11Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]:            |__Flow 'volume_create_manager': rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 849, in create_cloned_volume
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     src_volume.create_snap(clone_snap)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     result = task.execute(**arguments)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_image(context,
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return r(f, *args, **kwargs)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return fut.result()
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return self.__get_result()
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     raise self._exception
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     result = fn(*args, **kwargs)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_image_cache_or_download(
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update, cloned = self._create_from_image_cache(
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_source_volume(
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 860, in create_cloned_volume
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     src_volume.unprotect_snap(clone_snap)
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:12.455063 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:12.479617 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (77b13636-98ba-49b7-8b2b-18e2a69da3d4) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:12.485734 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (77b13636-98ba-49b7-8b2b-18e2a69da3d4) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:12.497200 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96953e14-d853-4bfb-a845-315c921f2c41) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:12.502618 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (96953e14-d853-4bfb-a845-315c921f2c41) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:12.510371 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c9b4d246-0961-4de9-a240-88d776f710dd) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:12.515464 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (c9b4d246-0961-4de9-a240-88d776f710dd) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:12.526925 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c27fcecc-3349-4a4a-9cb7-893b585748ee) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:12.530015 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Updating volume f054c8e1-9900-4dfe-8743-857b892350ae with {'scheduled_at': datetime.datetime(2022, 9, 24, 20, 57, 12, 529224), 'host': None}. {{(pid=148026) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}}
Sep 24 20:57:12.592284 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: re-scheduling SchedulerAPI.create_volume attempt 2 due to [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap') {{(pid=148026) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}}
Sep 24 20:57:12.646456 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.quota [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Created reservations ['61b56318-6250-4d17-92fe-26639e0cac43', 'b37254ed-95d1-4700-add4-ab994849662f', '25a8ebe5-8f32-45c1-8780-126ee82b3fa6', '513f1d8b-fdab-4825-ba78-49f4f7a84e2f'] {{(pid=148026) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 24 20:57:12.694843 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: re-scheduled {{(pid=148026) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}}
Sep 24 20:57:12.696403 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:12.849162 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] volume volume-f054c8e1-9900-4dfe-8743-857b892350ae no longer exists in backend
Sep 24 20:57:12.858197 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (c27fcecc-3349-4a4a-9cb7-893b585748ee) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True'
Sep 24 20:57:12.873916 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1f7b945d-dc10-4f61-9f71-b5a45a52b0bc) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:12.887198 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1f7b945d-dc10-4f61-9f71-b5a45a52b0bc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:12.901243 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Flow 'volume_create_manager' (0e3214e7-a70c-485f-8692-812ebb5fbdcd) transitioned into state 'REVERTED' from state 'RUNNING'
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Exception during message handling: rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 849, in create_cloned_volume
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     src_volume.create_snap(clone_snap)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     _run_flow()
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     flow_engine.run()
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     failures[0].reraise()
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 350, in reraise
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise value
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image(context,
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return fut.result()
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self.__get_result()
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise self._exception
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image_cache_or_download(
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update, cloned = self._create_from_image_cache(
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_source_volume(
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 860, in create_cloned_volume
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     src_volume.unprotect_snap(clone_snap)
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:12.909443 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:13.044317 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:13.144826 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']} {{(pid=148026) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1368}}
Sep 24 20:57:13.184570 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Flow 'volume_create_manager' (369ac8bb-15db-4d0b-b7ec-1bda5f34941f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 24 20:57:13.204557 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8675710-a201-4b11-b053-181a10c57c4b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:13.212183 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:13.331841 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.manager [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Notifying Schedulers of capabilities ... {{(pid=148026) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:57:13.357507 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8675710-a201-4b11-b053-181a10c57c4b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:13Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:13.383236 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a285b45a-e7b7-4b1c-90e4-0a6fcf0f29b2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:13.412086 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a285b45a-e7b7-4b1c-90e4-0a6fcf0f29b2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:13.428059 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b8cb666c-8723-4e12-99a7-93db35ea905d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:13.451545 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Deleted volume successfully.
Sep 24 20:57:13.490585 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-3ef70248-2e53-48c0-80b0-3d02bbd835f7 tempest-LiveMigrationTest-471200620 None] Lock "cinder-3c6e1dda-a103-4e0d-b86a-6da231607ef6-delete_volume" released by "delete_volume" :: held 1.721s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:57:13.617058 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b8cb666c-8723-4e12-99a7-93db35ea905d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628ab624c0>}' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:13.623004 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d963d11a-e6c3-4d50-aa49-4b985bae610d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:13.632653 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d963d11a-e6c3-4d50-aa49-4b985bae610d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 24 20:57:13.644118 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba8bb37c-866d-43d7-a457-d2dba22d1cbc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:13.645411 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628ab624c0>}
Sep 24 20:57:13.645847 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Cloning f054c8e1-9900-4dfe-8743-857b892350ae from image 4c30e8e9-8fb2-4377-83f1-98322cb17799  at location (None, None). {{(pid=148026) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1071}}
Sep 24 20:57:13.654968 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Acquiring lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" by "_prepare_image_cache_entry" {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 24 20:57:13.668455 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" acquired by "_prepare_image_cache_entry" :: waited 0.022s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 24 20:57:13.716403 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 57, 13, 697219)}. {{(pid=148026) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:57:13.716868 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=148026) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:57:13.717282 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=148026) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:886}}
Sep 24 20:57:13.724934 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.coordination [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Lock "cinder-4c30e8e9-8fb2-4377-83f1-98322cb17799" released by "_prepare_image_cache_entry" :: held 0.055s {{(pid=148026) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 24 20:57:13.728964 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Attempting to retrieve cache entry for image = 4c30e8e9-8fb2-4377-83f1-98322cb17799 on host ncdv-master-1@ceph#ceph. {{(pid=148026) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:827}}
Sep 24 20:57:13.749194 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Found image-volume cache entry: {'id': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'volume_id': 'a410930c-3d96-4d0b-b7f5-ee675704f604', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49), 'last_used': datetime.datetime(2022, 9, 24, 20, 57, 13, 743080)}. {{(pid=148026) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 24 20:57:13.749908 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.image.cache [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Image-volume cache entry image_update_at = 2022-09-24 20:40:49+00:00, requested image updated_at = 2022-09-24 20:40:49+00:00. {{(pid=148026) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 24 20:57:13.750586 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Creating from source image-volume a410930c-3d96-4d0b-b7f5-ee675704f604 {{(pid=148026) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:841}}
Sep 24 20:57:13.792963 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:13.958755 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] creating snapshot='volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' {{(pid=148026) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:846}}
Sep 24 20:57:13.989971 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba8bb37c-866d-43d7-a457-d2dba22d1cbc) transitioned into state 'FAILURE' from state 'RUNNING'
Sep 24 20:57:13.989971 ncdv-master-1 cinder-volume[148026]: 5 predecessors (most recent first):
Sep 24 20:57:13.989971 ncdv-master-1 cinder-volume[148026]:   Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f628b43a520>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:13Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}, 'provides': None}
Sep 24 20:57:13.989971 ncdv-master-1 cinder-volume[148026]:   |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f628b43a520>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:13Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8), 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=4c30e8e9-8fb2-4377-83f1-98322cb17799,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(f054c8e1-9900-4dfe-8743-857b892350ae),volume_id=f054c8e1-9900-4dfe-8743-857b892350ae,volume_properties=VolumeProperties,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_size': 1, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': 'a8f45f34e55f4f50b20d34824a4e60af', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'created_at': datetime.datetime(2022, 9, 24, 20, 40, 32, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 24, 20, 40, 49, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/4c30e8e9-8fb2-4377-83f1-98322cb17799/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f628ab624c0>}}
Sep 24 20:57:13.989971 ncdv-master-1 cinder-volume[148026]:      |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': '4c30e8e9-8fb2-4377-83f1-98322cb17799', 'snapshot_id': None, 'source_volid': None, 'volume_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume': {'id': 'f054c8e1-9900-4dfe-8743-857b892350ae', '_name_id': None, 'ec2_id': None, 'user_id': '56064ff9075243879d7bce37e3e547e7', 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'name': 'ceph', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-24T20:40:15.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2022-09-24T20:57:09.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-f054c8e1-9900-4dfe-8743-857b892350ae', 'name_id': 'f054c8e1-9900-4dfe-8743-857b892350ae', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': '291311aa-f438-45f1-8d44-6f2e6ad780f8', 'name': 'ceph', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-24T20:40:15.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'qos_specs': None, 'reservations': ['339bbf31-d5a1-48d8-83d6-ad840029e9a7', '0402076a-a418-41f6-a379-caf6d4d54aff', '9b0ddaf1-365a-4b76-8805-d629cff4e26a', '1c92821c-b0c3-441a-afc4-9e8553564953'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest-volume-332313835', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '45916bd2a2e847b5ae05290dd106b04c', 'qos_specs': None, 'reservations': ['339bbf31-d5a1-48d8-83d6-ad840029e9a7', '0402076a-a418-41f6-a379-caf6d4d54aff', '9b0ddaf1-365a-4b76-8805-d629cff4e26a', '1c92821c-b0c3-441a-afc4-9e8553564953'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'volume_type_id': '291311aa-f438-45f1-8d44-6f2e6ad780f8'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2022-09-24T20:40:15Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=291311aa-f438-45f1-8d44-6f2e6ad780f8,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2022-09-24T20:40:15Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=291311aa-f438-45f1-8d44-6f2e6ad780f8,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '56064ff9075243879d7bce37e3e547e7', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']}}, 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=4c30e8e9-8fb2-4377-83f1-98322cb17799,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(f054c8e1-9900-4dfe-8743-857b892350ae),volume_id=f054c8e1-9900-4dfe-8743-857b892350ae,volume_properties=VolumeProperties,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8)), 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:13Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8), 'context': <cinder.context.RequestContext object at 0x7f628b43a520>}, 'provides': None}
Sep 24 20:57:13.989971 ncdv-master-1 cinder-volume[148026]:         |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f628b43a520>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:13Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-24T20:57:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest-volume-332313835',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f054c8e1-9900-4dfe-8743-857b892350ae,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='45916bd2a2e847b5ae05290dd106b04c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-24T20:57:13Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-24T20:57:13Z,use_quota=True,user_id='56064ff9075243879d7bce37e3e547e7',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(291311aa-f438-45f1-8d44-6f2e6ad780f8),volume_type_id=291311aa-f438-45f1-8d44-6f2e6ad780f8)}
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]:            |__Flow 'volume_create_manager': rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 849, in create_cloned_volume
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     src_volume.create_snap(clone_snap)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     result = task.execute(**arguments)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_image(context,
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return r(f, *args, **kwargs)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return fut.result()
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     return self.__get_result()
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     raise self._exception
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     result = fn(*args, **kwargs)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_image_cache_or_download(
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update, cloned = self._create_from_image_cache(
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self._create_from_source_volume(
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 860, in create_cloned_volume
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager     src_volume.unprotect_snap(clone_snap)
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:13.990719 ncdv-master-1 cinder-volume[148026]: ERROR cinder.volume.manager 
Sep 24 20:57:14.017110 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba8bb37c-866d-43d7-a457-d2dba22d1cbc) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:14.033780 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (ba8bb37c-866d-43d7-a457-d2dba22d1cbc) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:14.050283 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d963d11a-e6c3-4d50-aa49-4b985bae610d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:14.054036 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (d963d11a-e6c3-4d50-aa49-4b985bae610d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:14.059017 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b8cb666c-8723-4e12-99a7-93db35ea905d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:14.063265 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b8cb666c-8723-4e12-99a7-93db35ea905d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:14.068807 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a285b45a-e7b7-4b1c-90e4-0a6fcf0f29b2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:14.072711 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Updating volume f054c8e1-9900-4dfe-8743-857b892350ae with {'scheduled_at': datetime.datetime(2022, 9, 24, 20, 57, 14, 69761), 'host': None}. {{(pid=148026) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:139}}
Sep 24 20:57:14.086002 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: re-scheduling SchedulerAPI.create_volume attempt 3 due to [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap') {{(pid=148026) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:162}}
Sep 24 20:57:14.118131 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.flows.manager.create_volume [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Volume f054c8e1-9900-4dfe-8743-857b892350ae: re-scheduled {{(pid=148026) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:179}}
Sep 24 20:57:14.119966 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:14.234244 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.drivers.rbd [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] volume volume-f054c8e1-9900-4dfe-8743-857b892350ae no longer exists in backend
Sep 24 20:57:14.243429 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (a285b45a-e7b7-4b1c-90e4-0a6fcf0f29b2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True'
Sep 24 20:57:14.269509 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8675710-a201-4b11-b053-181a10c57c4b) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=148026) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 24 20:57:14.280227 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (d8675710-a201-4b11-b053-181a10c57c4b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 24 20:57:14.305510 ncdv-master-1 cinder-volume[148026]: WARNING cinder.volume.manager [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Flow 'volume_create_manager' (369ac8bb-15db-4d0b-b7ec-1bda5f34941f) transitioned into state 'REVERTED' from state 'RUNNING'
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server [None req-2616fa9d-1a93-4270-a7ac-b6177476605a tempest-TaggedBootDevicesTest-1987726953 None] Exception during message handling: rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 849, in create_cloned_volume
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     src_volume.create_snap(clone_snap)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap' from b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604')
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     _run_flow()
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     flow_engine.run()
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     failures[0].reraise()
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 350, in reraise
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise value
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1237, in execute
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image(context,
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return fut.result()
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     return self.__get_result()
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     raise self._exception
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in _create_from_image
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image_cache_or_download(
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 944, in _create_from_image_cache_or_download
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update, cloned = self._create_from_image_cache(
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 843, in _create_from_image_cache
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_source_volume(
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 659, in _create_from_source_volume
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 860, in create_cloned_volume
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server     src_volume.unprotect_snap(clone_snap)
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-a410930c-3d96-4d0b-b7f5-ee675704f604'@b'volume-f054c8e1-9900-4dfe-8743-857b892350ae.clone_snap')
Sep 24 20:57:14.311484 ncdv-master-1 cinder-volume[148026]: ERROR oslo_messaging.rpc.server 
Sep 24 20:57:17.494634 ncdv-master-1 cinder-volume[148026]: DEBUG dbcounter [-] [148026] Writing DB stats cinder:SELECT=66,cinder:UPDATE=24,cinder:DELETE=3,cinder:INSERT=7 {{(pid=148026) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:57:22.142130 ncdv-master-1 cinder-volume[148026]: INFO cinder.volume.manager [req-dd28b781-a226-4f88-8606-c599d63bd824 req-f9dcc31a-c97f-4656-8b17-7370534d0238 tempest-LiveAutoBlockMigrationV225Test-1380422539 None] Terminate volume connection completed successfully.
Sep 24 20:57:22.143575 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.manager [req-dd28b781-a226-4f88-8606-c599d63bd824 req-f9dcc31a-c97f-4656-8b17-7370534d0238 tempest-LiveAutoBlockMigrationV225Test-1380422539 None] Deleting attachment b9160bb8-44bc-4581-9e42-20e1cdc137e6. {{(pid=148026) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 24 20:57:32.144078 ncdv-master-1 cinder-volume[148026]: DEBUG dbcounter [-] [148026] Writing DB stats cinder:SELECT=2 {{(pid=148026) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 24 20:57:37.302466 ncdv-master-1 cinder-volume[148026]: DEBUG oslo_service.periodic_task [None req-5ac98736-90e0-4219-9daa-6a00dac0600a None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=148026) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 24 20:57:37.303930 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-5ac98736-90e0-4219-9daa-6a00dac0600a None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:37.399242 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.volume.drivers.rbd [None req-5ac98736-90e0-4219-9daa-6a00dac0600a None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=148026) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:586}}
Sep 24 20:57:37.537166 ncdv-master-1 cinder-volume[148026]: DEBUG cinder.manager [None req-5ac98736-90e0-4219-9daa-6a00dac0600a None None] Notifying Schedulers of capabilities ... {{(pid=148026) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 24 20:57:37.546141 ncdv-master-1 cinder-volume[148026]: DEBUG dbcounter [-] [148026] Writing DB stats cinder:SELECT=64,cinder:UPDATE=30,cinder:INSERT=4 {{(pid=148026) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}