-- Logs begin at Sat 2022-09-03 20:01:17 UTC, end at Sat 2022-09-03 22:07:49 UTC. --
Sep 03 20:39:36.578980 ncdv-master-1 systemd[1]: Started Devstack devstack@c-vol.service.
Sep 03 20:39:39.966227 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "singleton_lock" {{(pid=133292) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:293}}
Sep 03 20:39:39.967148 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=133292) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:296}}
Sep 03 20:39:39.967148 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=133292) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:314}}
Sep 03 20:39:40.386518 ncdv-master-1 cinder-volume[133292]: INFO dbcounter [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Registered counter for database cinder
Sep 03 20:39:40.405589 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_db.sqlalchemy.engines [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133292) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 03 20:39:40.413338 ncdv-master-1 cinder-volume[133292]: DEBUG dbcounter [-] [133292] Writer thread running {{(pid=133292) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}}
Sep 03 20:39:40.721515 ncdv-master-1 cinder-volume[133292]: INFO cinder.rpc [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version.
Sep 03 20:39:40.726473 ncdv-master-1 cinder-volume[133292]: INFO cinder.rpc [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version.
Sep 03 20:39:40.733805 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_db.sqlalchemy.engines [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133292) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 03 20:39:40.803873 ncdv-master-1 cinder-volume[133292]: INFO cinder.volume.manager [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Determined volume DB was empty at startup.
Sep 03 20:39:40.804677 ncdv-master-1 cinder-volume[133292]: DEBUG cinder.volume.manager [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Cinder Volume DB check: vol_db_empty=True {{(pid=133292) __init__ /opt/stack/cinder/cinder/volume/manager.py:290}}
Sep 03 20:39:40.813985 ncdv-master-1 cinder-volume[133292]: INFO cinder.volume.manager [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Service not found for updating active_backend_id, assuming default for driver init.
Sep 03 20:39:40.814692 ncdv-master-1 cinder-volume[133292]: WARNING stevedore.named [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager
Sep 03 20:39:40.815141 ncdv-master-1 cinder-volume[133292]: WARNING castellan.key_manager [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
Sep 03 20:39:40.816279 ncdv-master-1 cinder-volume[133292]: WARNING cinder.keymgr.conf_key_mgr [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] This key manager is insecure and is not recommended for production deployments: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
Sep 03 20:39:40.833964 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_db.sqlalchemy.engines [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133292) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 03 20:39:40.840244 ncdv-master-1 cinder-volume[133292]: INFO cinder.rpc [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Automatically selected cinder-volume objects version 1.39 as minimum service version.
Sep 03 20:39:40.845123 ncdv-master-1 cinder-volume[133292]: INFO cinder.rpc [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version.
Sep 03 20:39:40.845976 ncdv-master-1 cinder-volume[133292]: WARNING stevedore.named [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager
Sep 03 20:39:40.846529 ncdv-master-1 cinder-volume[133292]: WARNING castellan.key_manager [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
Sep 03 20:39:40.848586 ncdv-master-1 cinder-volume[133292]: INFO cinder.volume.manager [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Image-volume cache enabled for host ncdv-master-1@ceph.
Sep 03 20:39:40.854978 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_db.sqlalchemy.engines [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133292) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 03 20:39:40.879621 ncdv-master-1 cinder-volume[133292]: INFO oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Starting 1 workers
Sep 03 20:39:40.884879 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Started child 133464 {{(pid=133292) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Sep 03 20:39:40.885948 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Full set of CONF: {{(pid=133292) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}}
Sep 03 20:39:40.886509 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ******************************************************************************** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}}
Sep 03 20:39:40.886986 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Configuration options gathered from: {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}}
Sep 03 20:39:40.887520 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}}
Sep 03 20:39:40.888330 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}}
Sep 03 20:39:40.888783 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ================================================================================ {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}}
Sep 03 20:39:40.889676 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] allow_availability_zone_fallback = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.890197 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] allow_compression_on_image_upload = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.890837 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] allowed_direct_url_schemes     = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.891407 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] api_rate_limit                 = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.892044 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] auth_strategy                  = keystone {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.892481 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] az_cache_duration              = 3600 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.893058 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_availability_zone      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.893172 ncdv-master-1 cinder-volume[133464]: INFO cinder.service [-] Starting cinder-volume node (version 20.1.0)
Sep 03 20:39:40.894139 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_host                   = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.894679 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_name                   = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.895272 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_stats_polling_interval = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.895810 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backup_api_class               = cinder.backup.api.API {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.896398 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backup_manager                 = cinder.backup.manager.BackupManager {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.896966 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backup_name_template           = backup-%s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.897586 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backup_use_same_host           = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.898107 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backup_use_temp_snapshot       = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.898714 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] chap_password                  = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.899833 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] chap_username                  =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.900369 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] chiscsi_conf                   = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.900989 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_internal_tenant_project_id = 1e040b2fca724c58a2660441a72cd350 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.901542 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_internal_tenant_user_id = 133a1ff64e224593be467eb7c27cbb09 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.902121 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cloned_volume_same_az          = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.902655 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cluster                        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.903225 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] compression_format             = gzip {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.903808 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] compute_api_class              = cinder.compute.nova.API {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.904479 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] config_dir                     = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.904832 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] config_file                    = ['/etc/cinder/cinder.conf'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.905259 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] config_source                  = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.905647 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] consistencygroup_api_class     = cinder.consistencygroup.api.API {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.906089 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] control_exchange               = openstack {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.906521 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] debug                          = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.906898 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] default_availability_zone      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.907256 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] default_group_type             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.907829 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.908423 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] default_volume_type            = ceph {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.909128 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] driver_client_cert             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.909656 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] driver_client_cert_key         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.910218 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] driver_data_namespace          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.910734 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] driver_ssl_cert_path           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.911255 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] driver_ssl_cert_verify         = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.911644 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] driver_use_ssl                 = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.912065 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] enable_force_upload            = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.912421 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] enable_new_services            = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.912784 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] enable_unsupported_driver      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.913197 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] enabled_backends               = ['ceph'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.913588 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] enforce_multipath_for_image_xfer = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.913934 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] filter_function                = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.914444 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_api_insecure            = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.915065 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_api_servers             = ['http://10.222.0.16/image'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.916088 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_api_ssl_compression     = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.916649 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_ca_certificates_file    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.917211 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_catalog_info            = image:glance:publicURL {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.917820 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_certfile                = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.918334 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_core_properties         = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.918711 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_keyfile                 = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.919101 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_num_retries             = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.919466 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] glance_request_timeout         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.919836 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] goodness_function              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.920216 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] graceful_shutdown_timeout      = 5 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.920609 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] group_api_class                = cinder.group.api.API {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.920943 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] host                           = ncdv-master-1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.921318 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iet_conf                       = /etc/iet/ietd.conf {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.929113 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_compress_on_upload       = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.929479 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_conversion_address_space_limit = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.929973 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_conversion_cpu_limit     = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.930666 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_conversion_dir           = /opt/stack/data/cinder/conversion {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.931140 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_conversion_disable       = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.931607 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_upload_use_cinder_backend = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.932070 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_upload_use_internal_tenant = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.932551 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_volume_cache_enabled     = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.933324 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_volume_cache_max_count   = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.933816 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] image_volume_cache_max_size_gb = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.934424 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Starting volume driver RBDDriver (1.2.0)
Sep 03 20:39:40.934582 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] init_host_max_objects_retrieval = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.935087 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] instance_format                = [instance: %(uuid)s]  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.935559 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] instance_uuid_format           = [instance: %(uuid)s]  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.936043 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iscsi_iotype                   = fileio {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.936365 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:40.936886 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iscsi_secondary_ip_addresses   = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.937539 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iscsi_target_flags             =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.938110 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iscsi_write_cache              = on {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.938785 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iser_helper                    = tgtadm {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.939756 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iser_ip_address                = 10.222.0.16 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.940381 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iser_port                      = 3260 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.940981 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] iser_target_prefix             = iqn.2010-10.org.openstack: {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.941574 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_config_append              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.942281 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_date_format                = %Y-%m-%d %H:%M:%S {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.943833 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_dir                        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.944627 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_file                       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.945193 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_options                    = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.945972 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_rotate_interval            = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.946838 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_rotate_interval_type       = days {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.947268 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] log_rotation_type              = none {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.947670 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] logging_context_format_string  = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.948071 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] logging_debug_format_suffix    = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.948552 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] logging_default_format_string  = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.949127 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] logging_exception_prefix       = ERROR %(name)s %(instance)s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.949625 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] logging_user_identity_format   = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.950115 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] max_age                        = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.950575 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] max_logfile_count              = 30 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.951039 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] max_logfile_size_mb            = 200 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.951539 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] max_over_subscription_ratio    = 20.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.952038 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] message_reap_interval          = 86400 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.952495 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] message_ttl                    = 2592000 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.953024 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] migration_create_volume_timeout_secs = 300 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.953501 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] monkey_patch                   = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.954175 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] monkey_patch_modules           = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.954594 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] my_ip                          = 10.222.0.16 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.955053 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] no_snapshot_gb_quota           = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.955508 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] num_iser_scan_tries            = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.955974 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] num_shell_tries                = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.956442 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] num_volume_device_scan_tries   = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.956887 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] nvmet_ns_id                    = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.957369 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] nvmet_port_id                  = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.957873 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_max_limit                = 1000 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.958380 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_volume_ext_list          = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.958889 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_volume_extension         = ['cinder.api.contrib.standard_extensions'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.959351 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_volume_listen            = 0.0.0.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.960164 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_volume_listen_port       = 8776 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.960629 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_volume_use_ssl           = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.961148 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] osapi_volume_workers           = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.961739 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] per_volume_size_limit          = -1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.963167 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] periodic_fuzzy_delay           = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.964683 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] periodic_interval              = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.965242 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] project_id_regex               = [0-9a-f\-]+ {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.965806 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] public_endpoint                = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.966382 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] publish_errors                 = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.967155 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_backup_gigabytes         = 1000 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.968713 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_backups                  = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.969333 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_consistencygroups        = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.969948 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_driver                   = cinder.quota.DbQuotaDriver {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.971202 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_gigabytes                = 1000 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.972300 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_groups                   = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.973473 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_snapshots                = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.974287 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] quota_volumes                  = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.974888 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] rate_limit_burst               = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.975581 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] rate_limit_except_level        = CRITICAL {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.976187 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] rate_limit_interval            = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.976762 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] reinit_driver_count            = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.977398 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] replication_device             = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.978496 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] report_discard_supported       = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.979085 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] report_interval                = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.979751 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] reservation_clean_interval     = 86400 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.980363 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] reservation_expire             = 86400 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.980951 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] reserved_percentage            = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.981843 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] resource_query_filters_file    = /etc/cinder/resource_filters.json {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.982314 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] rootwrap_config                = /etc/cinder/rootwrap.conf {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.982830 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] rpc_response_timeout           = 120 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.983307 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] run_external_periodic_tasks    = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.983774 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] scheduler_manager              = cinder.scheduler.manager.SchedulerManager {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.984236 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] scst_target_driver             = iscsi {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.984814 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] scst_target_iqn_name           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.985219 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_down_time              = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.985684 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] snapshot_name_template         = snapshot-%s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.986147 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] snapshot_same_host             = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.986615 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] split_loggers                  = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.987005 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] state_path                     = /opt/stack/data/cinder {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.987461 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] storage_availability_zone      = nova {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.987922 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] storage_protocol               = iSCSI {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.988401 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] syslog_log_facility            = LOG_USER {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.988859 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] target_helper                  = lioadm {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.989447 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] target_ip_address              = 10.222.0.16 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.989928 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] target_port                    = 3260 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.990392 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] target_prefix                  = iqn.2010-10.org.openstack: {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.990872 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] target_protocol                = iscsi {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.991447 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] trace_flags                    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.991910 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] transfer_api_class             = cinder.transfer.api.API {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.992383 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] transport_url                  = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.992832 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] until_refresh                  = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.993332 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_chap_auth                  = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.993782 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_default_quota_class        = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.994228 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_eventlog                   = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.994872 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_journal                    = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.995441 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_json                       = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.995801 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:40.996243 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_multipath_for_image_xfer   = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.996804 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_stderr                     = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.997386 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] use_syslog                     = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.998009 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] verify_glance_signatures       = enabled {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.998600 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_api_class               = cinder.volume.api.API {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.999162 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_backend_name            = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:40.999756 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_clear                   = zero {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.000316 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_clear_ionice            = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.000878 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_clear_size              = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.001600 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_copy_blkio_cgroup_name  = cinder-volume-copy {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.002165 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_copy_bps_limit          = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.002725 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_dd_blocksize            = 1M {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.003297 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_manager                 = cinder.volume.manager.VolumeManager {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.003867 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_name_template           = volume-%s {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.004446 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_service_inithost_offload = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.005044 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volume_usage_audit_period      = month {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.005690 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] volumes_dir                    = /opt/stack/data/cinder/volumes {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.006269 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] watch_log_file                 = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.006719 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] zoning_mode                    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}}
Sep 03 20:39:41.007311 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.007875 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.008390 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_concurrency.disable_process_locking = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.008938 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_concurrency.lock_path     = /opt/stack/data/cinder {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.009425 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.009877 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.010341 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_metrics.metrics_process_name =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.010810 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.011265 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.011899 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.admin_password = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.012357 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.012804 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.admin_token = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.013281 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.admin_user  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.013759 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_admin_prefix =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.014224 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_host   = 127.0.0.1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.014690 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_port   = 35357 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.015174 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_protocol = https {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.015682 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_section = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.016166 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_type   = password {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.016624 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_uri    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.017151 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.auth_version = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.017607 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.cache       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.018074 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.cafile      = /opt/stack/data/ca-bundle.pem {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.018565 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.certfile    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.019018 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.delay_auth_decision = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.019656 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.020122 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.http_connect_timeout = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.020574 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.021050 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.identity_uri = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.021771 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.include_service_catalog = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.023410 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.insecure    = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.023998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.interface   = public {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.025099 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.keyfile     = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.025912 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.026367 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.026797 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.027225 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.027657 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.028071 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.028526 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.035096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.035096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.035096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.region_name = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.035096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.035096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.service_token_roles_required = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.035096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.service_type = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.036998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.token_cache_time = 300 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.036998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.036998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.backend                  = dogpile.cache.null {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.036998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.backend_argument         = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.036998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.config_prefix            = cache.oslo {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.036998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.dead_timeout             = 60.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.037435 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.debug_cache_backend      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.037435 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.enable_retry_client      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.037435 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.enable_socket_keepalive  = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.037775 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.enabled                  = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.038723 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.expiration_time          = 600 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.039189 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.hashclient_retry_attempts = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.039636 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.hashclient_retry_delay   = 1.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.040632 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_dead_retry      = 300 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.041179 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_password        =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.044363 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.044981 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.045650 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_pool_maxsize    = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.046124 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.046674 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_sasl_enabled    = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.047158 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_servers         = ['localhost:11211'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.047618 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_socket_timeout  = 1.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.048063 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.memcache_username        =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.048522 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.proxies                  = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.048986 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.retry_attempts           = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.049432 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.retry_delay              = 0.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.050320 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.socket_keepalive_count   = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.050881 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.socket_keepalive_idle    = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.051391 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.socket_keepalive_interval = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.051809 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.tls_allowed_ciphers      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.052231 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.tls_cafile               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.052654 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.tls_certfile             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.053124 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.tls_enabled              = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.053558 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cache.tls_keyfile              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.054018 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] coordination.backend_url       = etcd3+http://10.222.0.16:2379 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.054400 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:41.054567 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.backend               = sqlalchemy {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.055079 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.connection            = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.055689 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.connection_debug      = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.056251 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.connection_parameters =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.056821 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.connection_recycle_time = 3600 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.057400 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.connection_trace      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.057947 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.db_inc_retry_interval = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.058501 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.db_max_retries        = 20 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.059044 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.db_max_retry_interval = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.059578 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.db_retry_interval     = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.060153 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.max_overflow          = 50 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.060699 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.max_pool_size         = 5 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.061296 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.max_retries           = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.061846 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.mysql_enable_ndb      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.062403 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.mysql_sql_mode        = TRADITIONAL {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.062939 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.pool_timeout          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.063527 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.retry_interval        = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.064126 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.slave_connection      = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.064669 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.sqlite_synchronous    = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.065237 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] database.use_db_reconnect      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.065839 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.enforce_new_defaults = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.066286 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.enforce_scope      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.066750 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.policy_default_rule = default {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.067203 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.policy_dirs        = ['policy.d'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.067671 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.policy_file        = policy.yaml {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.068150 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.068596 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.069046 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.069608 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.070125 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.070560 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.connection_string     = messaging:// {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.070860 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.enabled               = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.071748 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.es_doc_type           = notification {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.072120 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.es_scroll_size        = 10000 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.072484 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.es_scroll_time        = 2m {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.072835 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.filter_error_trace    = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.073228 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.hmac_keys             = SECRET_KEY {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.073600 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.sentinel_service_name = mymaster {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.073998 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.socket_timeout        = 0.1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.074360 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] profiler.trace_sqlalchemy      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.074738 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] os_brick.lock_path             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.075159 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] privsep_osbrick.capabilities   = [21] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.075533 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] privsep_osbrick.group          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.075880 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] privsep_osbrick.helper_command = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.076249 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] privsep_osbrick.logger_name    = oslo_privsep.daemon {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.076603 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.077010 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] privsep_osbrick.user           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.077411 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_sys_admin.capabilities  = [0, 1, 2, 3, 12, 21] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.077788 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_sys_admin.group         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.078134 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_sys_admin.helper_command = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.078499 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_sys_admin.logger_name   = oslo_privsep.daemon {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.078867 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.079204 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] cinder_sys_admin.user          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.079588 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.auth_section      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.080095 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.auth_type         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.080446 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.cafile            = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.080793 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.certfile          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.081198 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.collect_timing    = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.081558 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.insecure          = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.082090 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.keyfile           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.083124 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.send_service_user_token = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.083786 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.split_loggers     = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.084135 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] service_user.timeout           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.084852 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.backend_availability_zone = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.085231 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.backend_native_threads_pool_size = 20 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.085608 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.backup_use_temp_snapshot = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.085971 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.chap_password = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.086331 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.chap_username =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.086693 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.chiscsi_conf  = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.087599 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.087951 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.088297 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.driver_client_cert = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.090096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.driver_client_cert_key = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.090096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.driver_data_namespace = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.090096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.090096 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.090774 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.driver_use_ssl = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.090928 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.enable_deferred_deletion = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.091158 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.enable_unsupported_driver = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.091504 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.091889 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.extra_capabilities = {} {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.092238 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.filter_function = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.092581 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.goodness_function = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.092964 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iet_conf      = /etc/iet/ietd.conf {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.093317 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.093690 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.094471 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.094826 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.095189 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.095571 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iscsi_iotype  = fileio {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.096016 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.096475 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iscsi_target_flags =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.096877 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iscsi_write_cache = on {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.097898 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iser_helper   = tgtadm {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.098888 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iser_ip_address = 10.222.0.16 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.099266 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iser_port     = 3260 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.099659 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.100224 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.100750 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.101160 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.num_shell_tries = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.101521 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.101882 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.nvmet_ns_id   = 10 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.102240 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.nvmet_port_id = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.102634 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rados_connect_timeout = -1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.103006 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rados_connection_interval = 5 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.103370 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rados_connection_retries = 3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.103761 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_ceph_conf =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.104158 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_cluster_name = ceph {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.104515 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.104888 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.105419 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.106396 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_pool      = rbd {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.106921 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_secret_uuid = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.107457 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.107956 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.rbd_user      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.108658 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.replication_connect_timeout = 5 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.109319 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.replication_device = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.109930 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.report_discard_supported = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.110853 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.111479 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.reserved_percentage = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.112001 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.scst_target_driver = iscsi {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.112508 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.scst_target_iqn_name = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.113106 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.storage_protocol = iSCSI {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.113609 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.suppress_requests_ssl_warnings = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.114143 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.target_helper = tgtadm {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.114646 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.driver [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Driver hasn't implemented _init_vendor_properties()
Sep 03 20:39:41.114960 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.target_ip_address = 10.222.0.16 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.115204 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.driver [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Initialized capabilities list: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:35844ec5-7783-41e2-937f-44a580242021:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=133464) init_capabilities /opt/stack/cinder/cinder/volume/driver.py:812}}
Sep 03 20:39:41.115670 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.target_port   = 3260 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.116197 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.123048 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.target_protocol = iscsi {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.123667 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.trace_flags   = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.124226 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.unique_fqdn_network = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.124750 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.use_chap_auth = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.125300 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.125801 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_backend_name = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.126355 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_clear  = zero {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.126885 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_clear_ionice = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.127419 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_clear_size = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.127947 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.132602 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_db.sqlalchemy.engines [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133464) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 03 20:39:41.135406 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.135984 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.137743 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.138396 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] backend_defaults.volumes_dir   = /opt/stack/data/cinder/volumes {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.138963 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_reports.file_event_handler = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.139525 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.140067 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_reports.log_dir           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.140370 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writer thread running {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}}
Sep 03 20:39:41.140787 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.backend_availability_zone = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.141352 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.backend_host              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.141873 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.backend_native_threads_pool_size = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.142439 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.backup_use_temp_snapshot  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.142958 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.chap_password             = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.143475 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.chap_username             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.143974 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.chiscsi_conf              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.144490 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.deferred_deletion_delay   = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.145047 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.deferred_deletion_purge_interval = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.145564 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.driver_client_cert        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.146119 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.driver_client_cert_key    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.146624 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.driver_data_namespace     = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.147120 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.driver_ssl_cert_path      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.147730 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.driver_ssl_cert_verify    = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.148241 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.driver_use_ssl            = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.148741 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.enable_deferred_deletion  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.149329 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.enable_unsupported_driver = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.150353 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.enforce_multipath_for_image_xfer = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.150862 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.extra_capabilities        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.151371 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.filter_function           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.151880 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.goodness_function         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.152394 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iet_conf                  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.152919 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.image_upload_use_cinder_backend = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.153434 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.image_upload_use_internal_tenant = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.153969 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.image_volume_cache_enabled = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.154491 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.image_volume_cache_max_count = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.154991 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.image_volume_cache_max_size_gb = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.155500 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iscsi_iotype              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.156012 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iscsi_secondary_ip_addresses = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.156518 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iscsi_target_flags        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.157051 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iscsi_write_cache         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.157567 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iser_helper               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.158071 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iser_ip_address           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.158652 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iser_port                 = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.159157 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.iser_target_prefix        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.159666 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.max_over_subscription_ratio = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.160182 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.num_iser_scan_tries       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.160708 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.num_shell_tries           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.161235 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.num_volume_device_scan_tries = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.161743 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.nvmet_ns_id               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.162250 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.nvmet_port_id             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.162780 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rados_connect_timeout     = -1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.163343 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rados_connection_interval = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.163849 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rados_connection_retries  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.164372 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_ceph_conf             = /etc/ceph/ceph.conf {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.164925 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_cluster_name          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.165419 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_exclusive_cinder_pool = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.165957 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_flatten_volume_from_snapshot = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.166479 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_max_clone_depth       = 5 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.166988 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_pool                  = volumes {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.167495 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_secret_uuid           = ed553e1e-dd22-443c-962f-68bde7d14a09 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.168018 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_store_chunk_size      = 4 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.168544 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.rbd_user                  = cinder {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.169136 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.replication_connect_timeout = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.169651 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.replication_device        = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.170143 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.report_discard_supported  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.170637 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.report_dynamic_total_capacity = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.171156 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.reserved_percentage       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.171655 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.scst_target_driver        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.172147 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.scst_target_iqn_name      = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.172646 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.storage_protocol          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.173173 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.suppress_requests_ssl_warnings = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.173664 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.target_helper             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.174168 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.target_ip_address         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.174658 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.target_port               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.175146 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.target_prefix             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.175624 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.target_protocol           = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.176116 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.trace_flags               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.176627 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.unique_fqdn_network       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.177163 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.use_chap_auth             = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.177659 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.use_multipath_for_image_xfer = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.178186 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_backend_name       = ceph {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.178740 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_clear              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.179247 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_clear_ionice       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.179748 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_clear_size         = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.180249 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_copy_blkio_cgroup_name = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.180789 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_copy_bps_limit     = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.181363 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_dd_blocksize       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.181908 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volume_driver             = cinder.volume.drivers.rbd.RBDDriver {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.182422 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ceph.volumes_dir               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.183032 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.183558 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.184081 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.184636 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.185327 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.185903 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.186485 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.heartbeat_in_pthread = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.187034 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.187587 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.187931 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.db.sqlalchemy.api [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Building query based on filter {{(pid=133464) _process_snaps_filters /opt/stack/cinder/cinder/db/sqlalchemy/api.py:3885}}
Sep 03 20:39:41.188330 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.188922 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.189464 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.190040 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.190605 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.191163 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.191732 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.192288 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.192825 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.193458 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.194329 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_quroum_max_memory_bytes = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.195153 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_quroum_max_memory_length = 0 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.195981 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.197313 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.198132 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.198880 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.199469 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.ssl      = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.200072 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.ssl_ca_file =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.200642 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.ssl_cert_file =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.201438 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.202179 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.ssl_key_file =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.202750 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_rabbit.ssl_version =  {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.203313 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_notifications.driver = [] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.203875 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_notifications.retry = -1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.204667 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.205381 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] oslo_messaging_notifications.transport_url = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.205917 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] key_manager.backend            = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.206418 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] key_manager.fixed_key          = **** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.207086 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.auth_endpoint         = http://localhost/identity/v3 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.207808 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.barbican_api_version  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.208531 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.barbican_endpoint     = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.209151 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.barbican_endpoint_type = public {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.209948 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.barbican_region_name  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.210483 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.cafile                = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.211031 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.certfile              = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.211578 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.collect_timing        = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.212756 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.insecure              = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.213322 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.keyfile               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.213860 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.number_of_retries     = 60 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.214408 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.retry_delay           = 1 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.214916 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.image_utils [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Configuration image_conversion_dir is None or the path doesn't exist. {{(pid=133464) cleanup_temporary_file /opt/stack/cinder/cinder/image/image_utils.py:991}}
Sep 03 20:39:41.215125 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.send_service_user_token = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.215701 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.split_loggers         = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.216258 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.timeout               = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.216800 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.verify_ssl            = True {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.217359 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican.verify_ssl_path       = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.217925 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.auth_section = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.218485 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.auth_type = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.219028 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.cafile   = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.219552 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.certfile = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.220098 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.collect_timing = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.220711 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.insecure = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.221277 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.keyfile  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.221826 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.split_loggers = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.222358 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] barbican_service_user.timeout  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.222936 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.approle_role_id          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.223465 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.approle_secret_id        = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.223997 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.cafile                   = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.224543 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.certfile                 = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.225097 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.collect_timing           = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.225654 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.insecure                 = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.226193 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.keyfile                  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.226739 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.kv_mountpoint            = secret {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.227288 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.kv_version               = 2 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.227867 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.namespace                = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.228405 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.root_token_id            = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.228967 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.split_loggers            = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.229519 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.ssl_ca_crt_file          = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.230066 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.timeout                  = None {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.230319 ncdv-master-1 cinder-volume[133464]: INFO cinder.keymgr.migration [None req-8b8a0230-8aea-4854-b18d-f47b1fca98ab None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Sep 03 20:39:41.230813 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.use_ssl                  = False {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.231365 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] vault.vault_url                = http://127.0.0.1:8200 {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}}
Sep 03 20:39:41.231841 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] ******************************************************************************** {{(pid=133292) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}}
Sep 03 20:39:41.232375 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:41.262898 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:41.293542 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:39:41.323960 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Driver initialization completed successfully.
Sep 03 20:39:41.325123 ncdv-master-1 cinder-volume[133464]: INFO cinder.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Initiating service 3 cleanup
Sep 03 20:39:41.335660 ncdv-master-1 cinder-volume[133464]: INFO cinder.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Service 3 cleanup completed.
Sep 03 20:39:41.336075 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.service [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Creating RPC server for service cinder-volume {{(pid=133464) start /opt/stack/cinder/cinder/service.py:238}}
Sep 03 20:39:41.343989 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.service [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=133464) start /opt/stack/cinder/cinder/service.py:244}}
Sep 03 20:39:41.405766 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)
Sep 03 20:39:41.406695 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:41.443992 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:39:41.510737 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-ec6cbc65-bcb4-4fc2-9fef-439ceb7eceaf None None] Driver post RPC initialization completed successfully.
Sep 03 20:39:50.868131 ncdv-master-1 cinder-volume[133292]: DEBUG dbcounter [-] [133292] Writing DB stats cinder:SELECT=16,cinder:INSERT=1 {{(pid=133292) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:39:51.503809 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=25,cinder:INSERT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:39:51.506853 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:40:11.521973 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:40:21.528689 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:40:41.529114 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:40:41.530590 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:40:52.513955 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:40:52.514741 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:40:52.542852 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:40:52.571875 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:41:01.535615 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:41:11.544270 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:41:31.547950 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:41:41.555219 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:41:52.516866 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:41:52.517591 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:41:52.540498 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:41:52.564559 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:42:01.561523 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:42:21.559903 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:42:21.567053 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:42:41.568446 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:42:51.571330 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:42:52.519918 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:42:52.520568 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:42:52.546887 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:42:52.570561 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:43:01.580614 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:43:21.579533 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:43:21.582359 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:43:41.580426 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:43:51.592633 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:43:52.516112 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:43:52.516764 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:43:52.538096 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:43:52.560369 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:44:11.592164 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:44:31.599763 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:44:51.600579 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:44:52.516328 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:44:52.517002 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:44:52.538583 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:44:52.559934 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:45:01.609042 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:45:21.616875 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:45:31.620676 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:45:41.628724 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:45:52.514715 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:45:52.515386 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:45:52.537392 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:45:52.558887 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:46:01.624844 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:46:11.632375 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:46:21.642302 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:46:31.643754 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:46:51.649690 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:46:52.513346 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:46:52.514002 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:46:52.535952 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:46:52.558260 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:47:11.647254 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:47:11.648934 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:47:41.659096 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:47:41.659763 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:47:51.669276 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:47:52.512785 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:47:52.513467 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:47:52.536052 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:47:52.564084 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:48:21.666045 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:48:21.667675 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:48:41.678798 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:48:51.681765 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:48:52.508502 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:48:52.509720 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:48:52.588630 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:48:52.665575 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:49:01.684790 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:49:21.678681 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:49:41.688970 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:49:51.682045 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:49:52.511653 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:49:52.513779 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:49:52.580203 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:49:52.660614 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:50:01.695755 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:50:21.700797 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:UPDATE=3,cinder:SELECT=5 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:50:41.691031 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:50:52.510983 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:50:52.511755 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:50:52.587097 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:50:52.678181 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:51:01.709222 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:51:11.711662 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:51:31.719705 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:51:51.706716 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:51:52.515213 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:51:52.516116 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:51:52.626327 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:51:52.732261 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:52:11.719380 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:52:21.724190 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:52:41.710950 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:52:51.714311 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:52:52.510990 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:52:52.519286 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:52:52.600830 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:52:52.663262 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:53:01.720350 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:53:18.306767 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:53:18.353257 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Flow 'volume_create_manager' (c0a7e85f-d69f-413d-94a6-6d9754214698) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:53:18.359223 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0db0cd62-efed-44b4-bf0a-84f526bfbd0b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:53:18.411633 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (0db0cd62-efed-44b4-bf0a-84f526bfbd0b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:53:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-LiveMigrationTest-volume-689279915',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=80e6d713-e5fc-4d1b-ae13-80868b12d778,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6a7e10b9d8594d16a4c1eb63196ea329',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:53:18Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:53:18Z,use_quota=True,user_id='90d7e4c2e577467f949be3dc480cc7a2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:53:18.416715 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b26a6157-2489-4629-bfcb-5914b4b5f3dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:53:18.420557 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (b26a6157-2489-4629-bfcb-5914b4b5f3dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:53:18.425564 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3597ca4c-3ef0-4dfc-a9ba-84900d64beaf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:53:18.432519 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (3597ca4c-3ef0-4dfc-a9ba-84900d64beaf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '80e6d713-e5fc-4d1b-ae13-80868b12d778', 'volume_name': 'volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:53:18.435201 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fbf04d5-3f8b-462b-90af-ad8f9fdc6b82) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:53:18.438912 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (5fbf04d5-3f8b-462b-90af-ad8f9fdc6b82) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:53:18.443075 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (981f4fd4-825a-4f02-81a7-3989d9e1c66f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:53:18.444132 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Volume 80e6d713-e5fc-4d1b-ae13-80868b12d778: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'volume_size': 1}
Sep 03 20:53:18.444558 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] creating volume 'volume-80e6d713-e5fc-4d1b-ae13-80868b12d778' {{(pid=133464) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1040}}
Sep 03 20:53:18.445400 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:53:18.711424 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:53:18.712203 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:53:18.716063 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (981f4fd4-825a-4f02-81a7-3989d9e1c66f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:53:18.722960 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (12b0508c-139f-400b-abe7-8a0ce478cd9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:53:18.735859 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Volume volume-80e6d713-e5fc-4d1b-ae13-80868b12d778 (80e6d713-e5fc-4d1b-ae13-80868b12d778): created successfully
Sep 03 20:53:18.740726 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (12b0508c-139f-400b-abe7-8a0ce478cd9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:53:18.756536 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Flow 'volume_create_manager' (c0a7e85f-d69f-413d-94a6-6d9754214698) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:53:18.768351 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-5020d6df-98c9-4d53-b4b0-0d661b3adb94 tempest-LiveMigrationTest-2111888872 None] Created volume successfully.
Sep 03 20:53:20.716437 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-9abda89e-e2c9-49e3-9b30-7794d4954492 req-1d087952-c4bc-41d0-9127-6a714af7de70 tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:53:21.733934 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=13,cinder:UPDATE=5 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:53:22.364653 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-9abda89e-e2c9-49e3-9b30-7794d4954492 req-1d087952-c4bc-41d0-9127-6a714af7de70 tempest-LiveMigrationTest-2111888872 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.648s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:53:22.366398 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-9abda89e-e2c9-49e3-9b30-7794d4954492 req-1d087952-c4bc-41d0-9127-6a714af7de70 tempest-LiveMigrationTest-2111888872 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': '80e6d713-e5fc-4d1b-ae13-80868b12d778', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:53:22.517729 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-9abda89e-e2c9-49e3-9b30-7794d4954492 req-1d087952-c4bc-41d0-9127-6a714af7de70 tempest-LiveMigrationTest-2111888872 None] Connection info returned from driver {'name': 'volumes/volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '80e6d713-e5fc-4d1b-ae13-80868b12d778', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'f51d0dec-952e-4501-89b1-311bf26193e0'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:53:22.706039 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-9abda89e-e2c9-49e3-9b30-7794d4954492 req-1d087952-c4bc-41d0-9127-6a714af7de70 tempest-LiveMigrationTest-2111888872 None] attachment_update completed successfully.
Sep 03 20:53:41.730853 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=20,cinder:UPDATE=3,cinder:DELETE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:53:43.685845 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-20529a7b-3035-4d6c-80f2-9974b58c567e tempest-LiveMigrationTest-297325220 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:53:45.187326 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-20529a7b-3035-4d6c-80f2-9974b58c567e tempest-LiveMigrationTest-297325220 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.501s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:53:45.188783 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-20529a7b-3035-4d6c-80f2-9974b58c567e tempest-LiveMigrationTest-297325220 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': '80e6d713-e5fc-4d1b-ae13-80868b12d778', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:53:45.226871 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-20529a7b-3035-4d6c-80f2-9974b58c567e tempest-LiveMigrationTest-297325220 None] Connection info returned from driver {'name': 'volumes/volume-80e6d713-e5fc-4d1b-ae13-80868b12d778', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '80e6d713-e5fc-4d1b-ae13-80868b12d778', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '6cb8b898-b845-43c6-8ba8-529b6cb4e7de'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:53:45.343464 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-20529a7b-3035-4d6c-80f2-9974b58c567e tempest-LiveMigrationTest-297325220 None] attachment_update completed successfully.
Sep 03 20:54:01.738641 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=23,cinder:UPDATE=6 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:54:04.276973 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-6b2bac03-7633-4f41-be24-cd41509e4525 tempest-LiveMigrationTest-297325220 None] Terminate volume connection completed successfully.
Sep 03 20:54:04.278447 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-cc662109-9c16-44d9-ba2e-35cf35b4e275 req-6b2bac03-7633-4f41-be24-cd41509e4525 tempest-LiveMigrationTest-297325220 None] Deleting attachment f51d0dec-952e-4501-89b1-311bf26193e0. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:54:11.751109 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=22,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:54:19.026060 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-a0fd721a-4b3b-410f-88be-e90c73b125dd req-498b31dd-5621-4c72-9812-a5dac2dcb6cf tempest-LiveMigrationTest-2111888872 None] Terminate volume connection completed successfully.
Sep 03 20:54:19.027413 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-a0fd721a-4b3b-410f-88be-e90c73b125dd req-498b31dd-5621-4c72-9812-a5dac2dcb6cf tempest-LiveMigrationTest-2111888872 None] Deleting attachment 6cb8b898-b845-43c6-8ba8-529b6cb4e7de. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:54:22.555325 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:54:22.652611 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Flow 'volume_create_manager' (172e79c5-1ca5-4cb2-b8e0-5cd4fc0349c9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:54:22.666553 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (92f848cd-5eb8-45c0-bc83-58d543b680c0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:22.770108 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (92f848cd-5eb8-45c0-bc83-58d543b680c0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:54:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-LiveAutoBlockMigrationV225Test-volume-2013871386',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=7d6b5527-60b3-46e7-b592-f5108171cc0c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='361f907a187547aabda6cb54eeeb3684',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:54:22Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:54:23Z,use_quota=True,user_id='778561ee5e474accb31816f357985f44',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:22.788866 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7ea47803-0ec3-4cfc-8fc9-8b479e2dfe20) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:22.803990 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7ea47803-0ec3-4cfc-8fc9-8b479e2dfe20) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:22.833526 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6b9d61ae-45a2-40b2-b9b5-cbb1947f3159) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:22.845567 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (6b9d61ae-45a2-40b2-b9b5-cbb1947f3159) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '7d6b5527-60b3-46e7-b592-f5108171cc0c', 'volume_name': 'volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:22.855768 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6e1fa515-a170-4b66-8452-26a76c015083) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:22.861482 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (6e1fa515-a170-4b66-8452-26a76c015083) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:22.868974 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3afd17bf-0e60-4ac8-9361-a387ad7e31c5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:22.870127 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume 7d6b5527-60b3-46e7-b592-f5108171cc0c: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'volume_size': 1}
Sep 03 20:54:22.870859 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] creating volume 'volume-7d6b5527-60b3-46e7-b592-f5108171cc0c' {{(pid=133464) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1040}}
Sep 03 20:54:22.871694 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:54:23.010885 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:54:23.011294 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:54:23.015303 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (3afd17bf-0e60-4ac8-9361-a387ad7e31c5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:23.020013 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (97c65cec-b13b-42d7-b6e9-df8e16d690cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:23.056672 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:54:23.111953 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Flow 'volume_create_manager' (6e3b93ec-2371-4c68-991b-d1325264af28) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:54:23.131645 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e8a1fbd0-03d6-4d17-9a74-9720451c6c72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:23.179046 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e8a1fbd0-03d6-4d17-9a74-9720451c6c72) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:54:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-1896603214',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=3e825662-5770-41ff-ab3b-2e5005a78362,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='6a7e10b9d8594d16a4c1eb63196ea329',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:54:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:54:23Z,use_quota=True,user_id='90d7e4c2e577467f949be3dc480cc7a2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:23.197010 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (12ae612a-ff1f-40a4-9cf0-9d77bd3cca42) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:23.207326 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (12ae612a-ff1f-40a4-9cf0-9d77bd3cca42) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:23.223193 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e997251e-0bce-42a2-9eb6-fb0af834a73a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:23.268210 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume volume-7d6b5527-60b3-46e7-b592-f5108171cc0c (7d6b5527-60b3-46e7-b592-f5108171cc0c): created successfully
Sep 03 20:54:23.287674 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (97c65cec-b13b-42d7-b6e9-df8e16d690cf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:23.310668 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Flow 'volume_create_manager' (172e79c5-1ca5-4cb2-b8e0-5cd4fc0349c9) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:54:23.371626 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-bcf84a86-d2dc-47c5-a66f-0417cc2bca3c tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Created volume successfully.
Sep 03 20:54:23.442304 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (e997251e-0bce-42a2-9eb6-fb0af834a73a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '3e825662-5770-41ff-ab3b-2e5005a78362', 'volume_name': 'volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac962c70>}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:23.452154 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (79992cd3-a4b1-4331-996d-ad0b9c2daa35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:23.460286 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (79992cd3-a4b1-4331-996d-ad0b9c2daa35) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:54:23.473054 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83a6dcc5-513c-4c07-943b-ad61298f5d01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:54:23.480924 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume 3e825662-5770-41ff-ab3b-2e5005a78362: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac962c70>}
Sep 03 20:54:23.481869 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Cloning 3e825662-5770-41ff-ab3b-2e5005a78362 from image bb6b1dcc-96d7-4d20-8933-997caa341790  at location (None, None). {{(pid=133464) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1069}}
Sep 03 20:54:23.482973 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Acquiring lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" by "_prepare_image_cache_entry" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:54:23.502037 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" acquired by "_prepare_image_cache_entry" :: waited 0.019s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:54:23.525017 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Preparing cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=133464) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:889}}
Sep 03 20:54:23.526394 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Attempting to retrieve cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=133464) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:825}}
Sep 03 20:54:31.754366 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=16,cinder:UPDATE=5,cinder:DELETE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:54:35.742602 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-6524c721-4323-4efa-9ef6-77bf848fec5e req-048c5560-7dbd-4ce6-a459-0358f4f74dc8 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:37.295682 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-6524c721-4323-4efa-9ef6-77bf848fec5e req-048c5560-7dbd-4ce6-a459-0358f4f74dc8 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.553s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:37.296657 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-6524c721-4323-4efa-9ef6-77bf848fec5e req-048c5560-7dbd-4ce6-a459-0358f4f74dc8 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': '7d6b5527-60b3-46e7-b592-f5108171cc0c', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:54:37.337622 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-6524c721-4323-4efa-9ef6-77bf848fec5e req-048c5560-7dbd-4ce6-a459-0358f4f74dc8 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Connection info returned from driver {'name': 'volumes/volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '7d6b5527-60b3-46e7-b592-f5108171cc0c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '4ea47de2-f2ec-4824-adb4-002bdd484e18'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:54:37.415170 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-6524c721-4323-4efa-9ef6-77bf848fec5e req-048c5560-7dbd-4ce6-a459-0358f4f74dc8 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] attachment_update completed successfully.
Sep 03 20:54:40.090115 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Image fetch details: dest /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph, size 616.00 MB, duration 16.56 sec {{(pid=133464) fetch /opt/stack/cinder/cinder/image/image_utils.py:587}}
Sep 03 20:54:40.091795 ncdv-master-1 cinder-volume[133464]: INFO cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Image download 616.00 MB at 37.21 MB/s
Sep 03 20:54:40.188106 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:40.595421 ncdv-master-1 sudo[147177]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph
Sep 03 20:54:40.596331 ncdv-master-1 sudo[147177]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 03 20:54:42.473694 ncdv-master-1 sudo[147177]: pam_unix(sudo:session): session closed for user root
Sep 03 20:54:42.476221 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph" returned: 0 in 2.289s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:42.480140 ncdv-master-1 cinder-volume[133464]: WARNING py.warnings [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] /usr/local/lib/python3.8/dist-packages/oslo_utils/imageutils.py:71: FutureWarning: The human format is deprecated and the format parameter will be removed. Use explicitly json instead in version 'xena'
Sep 03 20:54:42.480140 ncdv-master-1 cinder-volume[133464]:   debtcollector.deprecate(
Sep 03 20:54:42.480140 ncdv-master-1 cinder-volume[133464]: 
Sep 03 20:54:42.481357 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Temporary image bb6b1dcc-96d7-4d20-8933-997caa341790 is fetched for user 90d7e4c2e577467f949be3dc480cc7a2. {{(pid=133464) fetch /opt/stack/cinder/cinder/image/image_utils.py:1124}}
Sep 03 20:54:42.548272 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=28,cinder:UPDATE=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:54:42.614677 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:42.880674 ncdv-master-1 sudo[147184]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph
Sep 03 20:54:42.883597 ncdv-master-1 sudo[147184]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 03 20:54:44.316787 ncdv-master-1 sudo[147184]: pam_unix(sudo:session): session closed for user root
Sep 03 20:54:44.320010 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph" returned: 0 in 1.706s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:44.321691 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] creating volume 'volume-3e825662-5770-41ff-ab3b-2e5005a78362' {{(pid=133464) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1040}}
Sep 03 20:54:44.323290 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:54:44.445513 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:54:44.446025 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:54:44.492620 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.volume_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Attempting download of bb6b1dcc-96d7-4d20-8933-997caa341790 ((None, None)) to volume 3e825662-5770-41ff-ab3b-2e5005a78362. {{(pid=133464) copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1190}}
Sep 03 20:54:44.574575 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_download_bb6b1dcc-96d7-4d20-8933-997caa341790_razubzej {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:44.767542 ncdv-master-1 sudo[147205]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_download_bb6b1dcc-96d7-4d20-8933-997caa341790_razubzej
Sep 03 20:54:44.769534 ncdv-master-1 sudo[147205]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 03 20:54:47.193089 ncdv-master-1 sudo[147205]: pam_unix(sudo:session): session closed for user root
Sep 03 20:54:47.197546 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_download_bb6b1dcc-96d7-4d20-8933-997caa341790_razubzej" returned: 0 in 2.628s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:47.200718 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:47.447916 ncdv-master-1 sudo[147221]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph
Sep 03 20:54:47.449021 ncdv-master-1 sudo[147221]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 03 20:54:49.753583 ncdv-master-1 sudo[147221]: pam_unix(sudo:session): session closed for user root
Sep 03 20:54:49.756362 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph" returned: 0 in 2.555s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:49.757873 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] bb6b1dcc-96d7-4d20-8933-997caa341790 was vhdx, converting to raw {{(pid=133464) fetch_to_volume_format /opt/stack/cinder/cinder/image/image_utils.py:825}}
Sep 03 20:54:49.759028 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -f vhdx /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph /opt/stack/data/cinder/conversion/tmpxozxay82 {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:49.821190 ncdv-master-1 sudo[147245]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -f vhdx /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph /opt/stack/data/cinder/conversion/tmpxozxay82
Sep 03 20:54:49.824671 ncdv-master-1 sudo[147245]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 03 20:54:50.692999 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-52ec2f5b-9f30-4129-8fa2-931d1e796cfa tempest-LiveAutoBlockMigrationV225Test-963097059 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:52.511385 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:54:52.513771 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:54:52.679417 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:54:52.810135 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:54:52.900970 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-52ec2f5b-9f30-4129-8fa2-931d1e796cfa tempest-LiveAutoBlockMigrationV225Test-963097059 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 2.208s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:52.901984 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-52ec2f5b-9f30-4129-8fa2-931d1e796cfa tempest-LiveAutoBlockMigrationV225Test-963097059 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': '7d6b5527-60b3-46e7-b592-f5108171cc0c', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:54:52.952824 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-52ec2f5b-9f30-4129-8fa2-931d1e796cfa tempest-LiveAutoBlockMigrationV225Test-963097059 None] Connection info returned from driver {'name': 'volumes/volume-7d6b5527-60b3-46e7-b592-f5108171cc0c', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '7d6b5527-60b3-46e7-b592-f5108171cc0c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '1815d6c7-8392-40fa-b4e8-c85247520fbd'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:54:53.115537 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-52ec2f5b-9f30-4129-8fa2-931d1e796cfa tempest-LiveAutoBlockMigrationV225Test-963097059 None] attachment_update completed successfully.
Sep 03 20:54:54.013252 ncdv-master-1 sudo[147245]: pam_unix(sudo:session): session closed for user root
Sep 03 20:54:54.036828 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -O raw -f vhdx /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph /opt/stack/data/cinder/conversion/tmpxozxay82" returned: 0 in 4.278s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:54.037774 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): /usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:54:54.289302 ncdv-master-1 sudo[147320]:   ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph
Sep 03 20:54:54.291126 ncdv-master-1 sudo[147320]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 03 20:54:56.345392 ncdv-master-1 sudo[147320]: pam_unix(sudo:session): session closed for user root
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "/usr/bin/python3.8 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph" returned: 0 in 2.310s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]: WARNING py.warnings [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] /usr/local/lib/python3.8/dist-packages/oslo_utils/imageutils.py:71: FutureWarning: The human format is deprecated and the format parameter will be removed. Use explicitly json instead in version 'xena'
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]:   debtcollector.deprecate(
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]: 
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Image conversion details: src /opt/stack/data/cinder/conversion/image_fetch_bb6b1dcc-96d7-4d20-8933-997caa341790_ntv5u5p9ncdv-master-1@ceph, size 700.00 MB, duration 4.28 sec, destination /opt/stack/data/cinder/conversion/tmpxozxay82 {{(pid=133464) _convert_image /opt/stack/cinder/cinder/image/image_utils.py:419}}
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]: INFO cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Converted 700.00 MB image at 163.60 MB/s
Sep 03 20:54:56.352695 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:54:56.451441 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] volume has no backup snaps {{(pid=133464) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1173}}
Sep 03 20:54:56.460352 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume volume-3e825662-5770-41ff-ab3b-2e5005a78362 is not a clone. {{(pid=133464) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1201}}
Sep 03 20:54:56.478027 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] deleting rbd volume volume-3e825662-5770-41ff-ab3b-2e5005a78362 {{(pid=133464) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1325}}
Sep 03 20:54:57.017397 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): rbd import --pool volumes --order 22 /opt/stack/data/cinder/conversion/tmpxozxay82 volume-3e825662-5770-41ff-ab3b-2e5005a78362 --new-format --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:55:01.811777 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=26,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:55:02.347294 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] CMD "rbd import --pool volumes --order 22 /opt/stack/data/cinder/conversion/tmpxozxay82 volume-3e825662-5770-41ff-ab3b-2e5005a78362 --new-format --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 5.330s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:55:02.530267 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:55:02.675643 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:55:02.676305 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:55:02.678561 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.volume_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Downloaded image bb6b1dcc-96d7-4d20-8933-997caa341790 ((None, None)) to volume 3e825662-5770-41ff-ab3b-2e5005a78362 successfully. {{(pid=133464) copy_image_to_volume /opt/stack/cinder/cinder/volume/volume_utils.py:1232}}
Sep 03 20:55:02.752919 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.image_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Temporary image bb6b1dcc-96d7-4d20-8933-997caa341790 for user 90d7e4c2e577467f949be3dc480cc7a2 is deleted. {{(pid=133464) fetch /opt/stack/cinder/cinder/image/image_utils.py:1128}}
Sep 03 20:55:02.815771 ncdv-master-1 cinder-volume[133464]: WARNING cinder.quota [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Deprecated: Default quota for resource: volumes_ceph is set by the default quota flag: quota_volumes_ceph, it is now deprecated. Please use the default quota class for default quota.
Sep 03 20:55:02.817272 ncdv-master-1 cinder-volume[133464]: WARNING cinder.quota [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Deprecated: Default quota for resource: gigabytes_ceph is set by the default quota flag: quota_gigabytes_ceph, it is now deprecated. Please use the default quota class for default quota.
Sep 03 20:55:02.986980 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.quota [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Created reservations ['7129dcff-708f-463f-861d-885012dfd9e0', 'd2237ea6-63ba-479f-b56b-27c0a5ff3c54', '96e757ae-661c-40dd-9806-a33aad85fcdc', '91a0a248-0cdc-46fc-bf30-e27be279d25d'] {{(pid=133464) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 03 20:55:03.018449 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Creating image volume entry: {'shared_targets': True, 'volume_glance_metadata': [{'key': 'signature_verified', 'value': 'False'}], 'service_uuid': None, 'provider_geometry': None, 'provider_auth': None, 'project_id': '1e040b2fca724c58a2660441a72cd350', 'replication_driver_data': None, 'host': 'ncdv-master-1@ceph#ceph', 'launched_at': None, 'ec2_id': None, 'volume_metadata': [], 'volume_admin_metadata': [], 'deleted': False, 'user_id': '90d7e4c2e577467f949be3dc480cc7a2', 'cluster_name': None, 'multiattach': False, 'created_at': datetime.datetime(2022, 9, 3, 20, 54, 22, tzinfo=datetime.timezone.utc), 'replication_status': None, 'encryption_key_id': None, 'metadata': {}, 'terminated_at': None, 'updated_at': datetime.datetime(2022, 9, 3, 20, 54, 23, tzinfo=datetime.timezone.utc), 'display_description': None, 'bootable': False, 'deleted_at': None, 'scheduled_at': datetime.datetime(2022, 9, 3, 20, 54, 23, tzinfo=datetime.timezone.utc), 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'availability_zone': 'nova', 'size': 1, 'consistencygroup_id': None, 'previous_status': None, 'provider_location': None, 'provider_id': None, 'group_id': None, 'replication_extended_status': None, 'display_name': 'image-bb6b1dcc-96d7-4d20-8933-997caa341790', 'attach_status': 'detached', 'status': 'creating', 'source_volid': '3e825662-5770-41ff-ab3b-2e5005a78362'}. {{(pid=133464) _clone_image_volume /opt/stack/cinder/cinder/volume/manager.py:1660}}
Sep 03 20:55:03.191738 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume reschedule parameters: False retry: None {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:55:03.253475 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Flow 'volume_create_manager' (7964c4f9-7764-4abc-bf4c-53e2b34224d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:55:03.264420 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e699e9a3-2321-4b56-8d1f-65c98054e3a3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:03.290104 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e699e9a3-2321-4b56-8d1f-65c98054e3a3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:54:22Z,deleted=False,deleted_at=None,display_description=None,display_name='image-bb6b1dcc-96d7-4d20-8933-997caa341790',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=2a98a68c-b86c-4af3-b4f2-5063b211791e,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1e040b2fca724c58a2660441a72cd350',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:54:23Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=3e825662-5770-41ff-ab3b-2e5005a78362,status='creating',terminated_at=None,updated_at=2022-09-03T20:54:23Z,use_quota=True,user_id='90d7e4c2e577467f949be3dc480cc7a2',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:03.297554 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f4850b0e-fae9-41ad-9876-dfb58cc52c53) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:03.302937 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (f4850b0e-fae9-41ad-9876-dfb58cc52c53) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:03.310006 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (29dcba37-178b-4246-bcf8-52b36b427deb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:03.337027 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (29dcba37-178b-4246-bcf8-52b36b427deb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'source_vol', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'volume_name': 'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e', 'volume_size': 1, 'source_volid': '3e825662-5770-41ff-ab3b-2e5005a78362', 'source_volstatus': 'downloading'}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:03.343684 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1762c0b5-7dce-416b-84c9-19643b6d2928) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:03.357421 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (1762c0b5-7dce-416b-84c9-19643b6d2928) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:03.366302 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (05676c1c-d344-4a45-8d92-d076dcaa724a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:03.375198 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume 2a98a68c-b86c-4af3-b4f2-5063b211791e: being created as source_vol with specification: {'status': 'creating', 'volume_name': 'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e', 'volume_size': 1, 'source_volid': '3e825662-5770-41ff-ab3b-2e5005a78362', 'source_volstatus': 'downloading'}
Sep 03 20:55:03.396687 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:55:03.485134 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] creating snapshot='volume-2a98a68c-b86c-4af3-b4f2-5063b211791e.clone_snap' {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:779}}
Sep 03 20:55:03.752436 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] cloning 'volume-3e825662-5770-41ff-ab3b-2e5005a78362@volume-2a98a68c-b86c-4af3-b4f2-5063b211791e.clone_snap' to 'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e' {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:785}}
Sep 03 20:55:03.938808 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume volume-3e825662-5770-41ff-ab3b-2e5005a78362 is not a clone. {{(pid=133464) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1201}}
Sep 03 20:55:03.940342 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:55:03.940928 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:55:03.979185 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] clone created successfully {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:858}}
Sep 03 20:55:03.983918 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (05676c1c-d344-4a45-8d92-d076dcaa724a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e', 'volume_size': 1, 'source_volid': '3e825662-5770-41ff-ab3b-2e5005a78362', 'source_volstatus': 'downloading'}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:03.989001 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (9382b5cb-fb20-44d9-b08c-3284ff1d9cc3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:04.015964 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume volume-2a98a68c-b86c-4af3-b4f2-5063b211791e (2a98a68c-b86c-4af3-b4f2-5063b211791e): created successfully
Sep 03 20:55:04.026183 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (9382b5cb-fb20-44d9-b08c-3284ff1d9cc3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:04.048792 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Flow 'volume_create_manager' (7964c4f9-7764-4abc-bf4c-53e2b34224d2) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:55:04.067549 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Created volume successfully.
Sep 03 20:55:04.138775 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Creating new image-volume cache entry for image bb6b1dcc-96d7-4d20-8933-997caa341790 on ncdv-master-1@ceph#ceph {{(pid=133464) create_cache_entry /opt/stack/cinder/cinder/image/cache.py:104}}
Sep 03 20:55:04.154737 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] New image-volume cache entry created: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 55, 4, 144680)}. {{(pid=133464) create_cache_entry /opt/stack/cinder/cinder/image/cache.py:129}}
Sep 03 20:55:04.162728 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" released by "_prepare_image_cache_entry" :: held 40.661s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:55:04.164100 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.volume_utils [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Marking volume 3e825662-5770-41ff-ab3b-2e5005a78362 as bootable. {{(pid=133464) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}}
Sep 03 20:55:04.178251 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Copying metadata from image bb6b1dcc-96d7-4d20-8933-997caa341790 to 3e825662-5770-41ff-ab3b-2e5005a78362. {{(pid=133464) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:451}}
Sep 03 20:55:04.178968 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Creating volume glance metadata for volume 3e825662-5770-41ff-ab3b-2e5005a78362 backed by image bb6b1dcc-96d7-4d20-8933-997caa341790 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': '', 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_name': 'ubuntu-16', 'checksum': '318f990048c8a194829e52141534610d', 'container_format': 'bare', 'disk_format': 'vhdx', 'min_disk': 0, 'min_ram': 0, 'size': 645922816}. {{(pid=133464) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:681}}
Sep 03 20:55:04.291245 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (83a6dcc5-513c-4c07-943b-ad61298f5d01) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac962c70>}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:04.297708 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (b9cb7c78-dcf3-4b05-97f5-d126e568721f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:04.311954 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Volume volume-3e825662-5770-41ff-ab3b-2e5005a78362 (3e825662-5770-41ff-ab3b-2e5005a78362): created successfully
Sep 03 20:55:04.320069 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (b9cb7c78-dcf3-4b05-97f5-d126e568721f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:04.331863 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Flow 'volume_create_manager' (6e3b93ec-2371-4c68-991b-d1325264af28) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:55:04.362744 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-1a0b1e1d-2828-4591-a8f5-ce55da8f917b tempest-LiveMigrationTest-2111888872 None] Created volume successfully.
Sep 03 20:55:09.580926 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-ee263f39-3874-4d66-a146-8cec265857b6 req-a06afd03-27e9-42e2-a991-ecc28970bc24 tempest-LiveMigrationTest-2111888872 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:55:11.636278 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-ee263f39-3874-4d66-a146-8cec265857b6 req-a06afd03-27e9-42e2-a991-ecc28970bc24 tempest-LiveMigrationTest-2111888872 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 2.055s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:55:11.637104 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-ee263f39-3874-4d66-a146-8cec265857b6 req-a06afd03-27e9-42e2-a991-ecc28970bc24 tempest-LiveMigrationTest-2111888872 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': '3e825662-5770-41ff-ab3b-2e5005a78362', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:55:11.663665 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=66,cinder:UPDATE=17,cinder:INSERT=15,cinder:DELETE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:55:11.697706 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-ee263f39-3874-4d66-a146-8cec265857b6 req-a06afd03-27e9-42e2-a991-ecc28970bc24 tempest-LiveMigrationTest-2111888872 None] Connection info returned from driver {'name': 'volumes/volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '3e825662-5770-41ff-ab3b-2e5005a78362', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'd80a84b3-10ca-4238-8858-17ffebdce6ba'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:55:11.798628 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-ee263f39-3874-4d66-a146-8cec265857b6 req-a06afd03-27e9-42e2-a991-ecc28970bc24 tempest-LiveMigrationTest-2111888872 None] attachment_update completed successfully.
Sep 03 20:55:12.878366 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-9510ee21-3b40-4c03-850b-3d0a7eb544d9 tempest-LiveAutoBlockMigrationV225Test-963097059 None] Terminate volume connection completed successfully.
Sep 03 20:55:12.879856 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-11a93b25-f8bb-45f3-a022-531cb334f58d req-9510ee21-3b40-4c03-850b-3d0a7eb544d9 tempest-LiveAutoBlockMigrationV225Test-963097059 None] Deleting attachment 4ea47de2-f2ec-4824-adb4-002bdd484e18. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:55:25.955860 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-84c4c8bc-9dbd-4f15-8ff4-f23f60f569eb req-71e64355-ffc0-46eb-a7dc-a8d6b8bed8a4 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Terminate volume connection completed successfully.
Sep 03 20:55:25.956666 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-84c4c8bc-9dbd-4f15-8ff4-f23f60f569eb req-71e64355-ffc0-46eb-a7dc-a8d6b8bed8a4 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Deleting attachment 1815d6c7-8392-40fa-b4e8-c85247520fbd. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:55:28.089703 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:55:28.116008 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Flow 'volume_create_manager' (c2a0d9f9-1c1d-4eb7-a437-0acb881a390d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:55:28.123533 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2f6602b6-502b-4be1-9bd0-8db7d5c9afad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:28.175547 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (2f6602b6-502b-4be1-9bd0-8db7d5c9afad) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:55:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-1598267494',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=f6866eae-8b17-4560-881a-3622ae0208ea,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='361f907a187547aabda6cb54eeeb3684',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:55:28Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:55:28Z,use_quota=True,user_id='778561ee5e474accb31816f357985f44',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:28.194399 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (81020501-2bd0-4b5c-b3cf-ffb16b88c249) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:28.204609 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (81020501-2bd0-4b5c-b3cf-ffb16b88c249) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:28.216554 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (137a42a6-efd5-4260-b2f6-d558fee4ba3b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:31.278259 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (137a42a6-efd5-4260-b2f6-d558fee4ba3b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'f6866eae-8b17-4560-881a-3622ae0208ea', 'volume_name': 'volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac819130>}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:31.284467 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (16a7d6ed-b4a1-4589-ba34-526a6d381cc7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:31.319611 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (16a7d6ed-b4a1-4589-ba34-526a6d381cc7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:31.337260 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8420131d-3106-48e2-a922-d92acdcd4290) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:31.342978 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume f6866eae-8b17-4560-881a-3622ae0208ea: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac819130>}
Sep 03 20:55:31.350067 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Cloning f6866eae-8b17-4560-881a-3622ae0208ea from image bb6b1dcc-96d7-4d20-8933-997caa341790  at location (None, None). {{(pid=133464) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1069}}
Sep 03 20:55:31.355942 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Acquiring lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" by "_prepare_image_cache_entry" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:55:31.387792 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" acquired by "_prepare_image_cache_entry" :: waited 0.032s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:55:31.416233 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 55, 31, 403582)}. {{(pid=133464) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:55:31.417097 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=133464) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:55:31.417816 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Found cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=133464) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:884}}
Sep 03 20:55:31.425840 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" released by "_prepare_image_cache_entry" :: held 0.038s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:55:31.427742 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Attempting to retrieve cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=133464) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:825}}
Sep 03 20:55:31.440173 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 55, 31, 435999)}. {{(pid=133464) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:55:31.440779 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=133464) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:55:31.441438 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Creating from source image-volume 2a98a68c-b86c-4af3-b4f2-5063b211791e {{(pid=133464) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:839}}
Sep 03 20:55:31.476455 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:55:31.619029 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] creating snapshot='volume-f6866eae-8b17-4560-881a-3622ae0208ea.clone_snap' {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:779}}
Sep 03 20:55:32.682022 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] cloning 'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e@volume-f6866eae-8b17-4560-881a-3622ae0208ea.clone_snap' to 'volume-f6866eae-8b17-4560-881a-3622ae0208ea' {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:785}}
Sep 03 20:55:32.719031 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=58,cinder:INSERT=9,cinder:UPDATE=12 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:55:32.865619 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume volume-3e825662-5770-41ff-ab3b-2e5005a78362 is not a clone. {{(pid=133464) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1201}}
Sep 03 20:55:32.866979 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:55:32.867562 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:55:32.909663 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] clone created successfully {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:858}}
Sep 03 20:55:32.910951 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.volume_utils [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Marking volume f6866eae-8b17-4560-881a-3622ae0208ea as bootable. {{(pid=133464) enable_bootable_flag /opt/stack/cinder/cinder/volume/volume_utils.py:1140}}
Sep 03 20:55:32.920653 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Copying metadata from image bb6b1dcc-96d7-4d20-8933-997caa341790 to f6866eae-8b17-4560-881a-3622ae0208ea. {{(pid=133464) _handle_bootable_volume_glance_meta /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:451}}
Sep 03 20:55:32.921387 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Creating volume glance metadata for volume f6866eae-8b17-4560-881a-3622ae0208ea backed by image bb6b1dcc-96d7-4d20-8933-997caa341790 with: {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': '', 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_name': 'ubuntu-16', 'checksum': '318f990048c8a194829e52141534610d', 'container_format': 'bare', 'disk_format': 'vhdx', 'min_disk': 0, 'min_ram': 0, 'size': 645922816}. {{(pid=133464) _capture_volume_image_metadata /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:681}}
Sep 03 20:55:33.116518 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (8420131d-3106-48e2-a922-d92acdcd4290) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac819130>}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:33.148922 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (ceb63759-9fb7-48a8-9055-733bb0406510) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:55:33.187114 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume volume-f6866eae-8b17-4560-881a-3622ae0208ea (f6866eae-8b17-4560-881a-3622ae0208ea): created successfully
Sep 03 20:55:33.191760 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (ceb63759-9fb7-48a8-9055-733bb0406510) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:55:33.198136 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Flow 'volume_create_manager' (c2a0d9f9-1c1d-4eb7-a437-0acb881a390d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:55:33.220781 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-f3a3b2bb-63ee-48ac-b63d-a953e3e338d6 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Created volume successfully.
Sep 03 20:55:40.618971 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-9de2a44f-00eb-4986-a46c-05c5154aaa7a req-5f01ba4c-f75b-4e91-8bfb-c1ae661c4eeb tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:55:42.253029 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-9de2a44f-00eb-4986-a46c-05c5154aaa7a req-5f01ba4c-f75b-4e91-8bfb-c1ae661c4eeb tempest-LiveAutoBlockMigrationV225Test-1007400564 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.634s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:55:42.257712 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-9de2a44f-00eb-4986-a46c-05c5154aaa7a req-5f01ba4c-f75b-4e91-8bfb-c1ae661c4eeb tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': 'f6866eae-8b17-4560-881a-3622ae0208ea', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:55:42.282264 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:UPDATE=6,cinder:SELECT=32,cinder:INSERT=12,cinder:DELETE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:55:42.304964 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-9de2a44f-00eb-4986-a46c-05c5154aaa7a req-5f01ba4c-f75b-4e91-8bfb-c1ae661c4eeb tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Connection info returned from driver {'name': 'volumes/volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'f6866eae-8b17-4560-881a-3622ae0208ea', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'e48c8a8a-e7b3-4d42-836c-e8aa5d0bc102'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:55:42.409885 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-9de2a44f-00eb-4986-a46c-05c5154aaa7a req-5f01ba4c-f75b-4e91-8bfb-c1ae661c4eeb tempest-LiveAutoBlockMigrationV225Test-1007400564 None] attachment_update completed successfully.
Sep 03 20:55:42.789126 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-412ff94f-f1e8-42c0-b88d-57562e2fd6d3 tempest-LiveMigrationTest-297325220 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:55:44.349347 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-412ff94f-f1e8-42c0-b88d-57562e2fd6d3 tempest-LiveMigrationTest-297325220 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.569s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:55:44.354795 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-412ff94f-f1e8-42c0-b88d-57562e2fd6d3 tempest-LiveMigrationTest-297325220 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': '3e825662-5770-41ff-ab3b-2e5005a78362', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:55:44.439945 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-412ff94f-f1e8-42c0-b88d-57562e2fd6d3 tempest-LiveMigrationTest-297325220 None] Connection info returned from driver {'name': 'volumes/volume-3e825662-5770-41ff-ab3b-2e5005a78362', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '3e825662-5770-41ff-ab3b-2e5005a78362', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'd46f3938-1515-460b-b796-5ba467eb50a7'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:55:44.572030 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-412ff94f-f1e8-42c0-b88d-57562e2fd6d3 tempest-LiveMigrationTest-297325220 None] attachment_update completed successfully.
Sep 03 20:55:52.516201 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:55:52.520747 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:55:52.597359 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:55:52.641268 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:55:59.020802 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-36379ac6-80f1-415d-ad7f-4dbecdbc6165 tempest-LiveMigrationTest-297325220 None] Terminate volume connection completed successfully.
Sep 03 20:55:59.021442 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-f7d37b90-7951-4b9b-91dd-6acdc73caac7 req-36379ac6-80f1-415d-ad7f-4dbecdbc6165 tempest-LiveMigrationTest-297325220 None] Deleting attachment d80a84b3-10ca-4238-8858-17ffebdce6ba. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:56:02.702567 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=22,cinder:UPDATE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:56:12.696134 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=35,cinder:UPDATE=8 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:56:14.264605 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-c310a8b2-b56c-445e-96fe-c6384cb4ed90 tempest-LiveMigrationTest-2111888872 None] Terminate volume connection completed successfully.
Sep 03 20:56:14.265339 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-c310a8b2-b56c-445e-96fe-c6384cb4ed90 tempest-LiveMigrationTest-2111888872 None] Deleting attachment d46f3938-1515-460b-b796-5ba467eb50a7. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:56:14.567714 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Acquiring lock "cinder-3e825662-5770-41ff-ab3b-2e5005a78362-delete_volume" by "delete_volume" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:56:14.584649 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Lock "cinder-3e825662-5770-41ff-ab3b-2e5005a78362-delete_volume" acquired by "delete_volume" :: waited 0.017s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:56:14.640654 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:14.743020 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] volume has no backup snaps {{(pid=133464) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1173}}
Sep 03 20:56:14.744068 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] volume has clone snapshot(s) {{(pid=133464) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1288}}
Sep 03 20:56:14.746000 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Volume volume-3e825662-5770-41ff-ab3b-2e5005a78362 is not a clone. {{(pid=133464) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1201}}
Sep 03 20:56:14.911748 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.quota [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Created reservations ['333fd4bb-0665-46c5-8a14-f6e411a0f656', '34c952fd-b9a3-491d-b9f4-724d2256dc13', 'f65ef488-702e-4d86-914d-cc7407a06300', '70dbc185-040b-40c1-999f-60fc5a271562'] {{(pid=133464) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 03 20:56:15.015764 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:15.073581 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:15.115904 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:56:15.120614 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Deleted volume successfully.
Sep 03 20:56:15.142909 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [req-7421f368-cabd-4c6e-bbd3-c0b84484c367 req-fb4827f2-12bb-4c6f-8ef2-0778e844f81c tempest-LiveMigrationTest-2111888872 None] Lock "cinder-3e825662-5770-41ff-ab3b-2e5005a78362-delete_volume" released by "delete_volume" :: held 0.558s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:56:16.022150 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Acquiring lock "cinder-80e6d713-e5fc-4d1b-ae13-80868b12d778-delete_volume" by "delete_volume" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:56:16.052792 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Lock "cinder-80e6d713-e5fc-4d1b-ae13-80868b12d778-delete_volume" acquired by "delete_volume" :: waited 0.033s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:56:16.171988 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:16.246714 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] volume has no backup snaps {{(pid=133464) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1173}}
Sep 03 20:56:16.246947 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Volume volume-80e6d713-e5fc-4d1b-ae13-80868b12d778 is not a clone. {{(pid=133464) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1201}}
Sep 03 20:56:16.255235 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] deleting rbd volume volume-80e6d713-e5fc-4d1b-ae13-80868b12d778 {{(pid=133464) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1325}}
Sep 03 20:56:16.676113 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.quota [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Created reservations ['f1ef6fbd-8d8d-4c12-bb41-9f9c59675fe3', 'fd892a47-a621-4e07-8a65-3eb6b7572008', 'a4fce7b2-dd46-4fb5-8831-fd4b47cf80c1', '7f30b22c-4b2c-4504-832c-5aa13ef8cc88'] {{(pid=133464) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 03 20:56:16.766721 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:16.851381 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:16.922603 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:56:16.929587 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Deleted volume successfully.
Sep 03 20:56:16.942753 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-19fa62eb-483c-423f-9f39-90eb463f5722 tempest-LiveMigrationTest-2111888872 None] Lock "cinder-80e6d713-e5fc-4d1b-ae13-80868b12d778-delete_volume" released by "delete_volume" :: held 0.890s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:56:20.378694 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-05ef022f-9f57-4a67-862b-296d4f8bbed0 tempest-LiveAutoBlockMigrationV225Test-963097059 None] Running cmd (subprocess): ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Sep 03 20:56:21.402086 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_concurrency.processutils [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-05ef022f-9f57-4a67-862b-296d4f8bbed0 tempest-LiveAutoBlockMigrationV225Test-963097059 None] CMD "ceph mon dump --format=json --id cinder --cluster ceph --conf /etc/ceph/ceph.conf" returned: 0 in 1.023s {{(pid=133464) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
Sep 03 20:56:21.403780 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-05ef022f-9f57-4a67-862b-296d4f8bbed0 tempest-LiveAutoBlockMigrationV225Test-963097059 None] connection data: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'ed553e1e-dd22-443c-962f-68bde7d14a09', 'volume_id': 'f6866eae-8b17-4560-881a-3622ae0208ea', 'discard': True}} {{(pid=133464) initialize_connection /opt/stack/cinder/cinder/volume/drivers/rbd.py:1649}}
Sep 03 20:56:21.449507 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-05ef022f-9f57-4a67-862b-296d4f8bbed0 tempest-LiveAutoBlockMigrationV225Test-963097059 None] Connection info returned from driver {'name': 'volumes/volume-f6866eae-8b17-4560-881a-3622ae0208ea', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'f6866eae-8b17-4560-881a-3622ae0208ea', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '8736066a-4b1f-45d8-a8e4-879ed457450f'} {{(pid=133464) _connection_create /opt/stack/cinder/cinder/volume/manager.py:4867}}
Sep 03 20:56:21.529368 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-05ef022f-9f57-4a67-862b-296d4f8bbed0 tempest-LiveAutoBlockMigrationV225Test-963097059 None] attachment_update completed successfully.
Sep 03 20:56:31.529634 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=36,cinder:UPDATE=16,cinder:INSERT=5 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:56:34.777901 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=47,cinder:UPDATE=21,cinder:INSERT=3,cinder:DELETE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:56:34.796139 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-b3dbe940-bb14-4efa-a95a-c7c487ce7452 tempest-LiveAutoBlockMigrationV225Test-963097059 None] Terminate volume connection completed successfully.
Sep 03 20:56:34.796589 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-38ede3e8-32ac-4ed4-ad9e-5ca64948f9ae req-b3dbe940-bb14-4efa-a95a-c7c487ce7452 tempest-LiveAutoBlockMigrationV225Test-963097059 None] Deleting attachment e48c8a8a-e7b3-4d42-836c-e8aa5d0bc102. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:56:44.792546 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:56:47.048473 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:56:47.147953 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Flow 'volume_create_manager' (8d8a39eb-8487-42f7-814b-3b5c68130fa9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:56:47.183832 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ad27258-3159-4c7e-b580-db19573d9115) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:56:47.234544 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (8ad27258-3159-4c7e-b580-db19573d9115) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:56:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-857615876',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=0adb6e45-9611-454a-af2d-365a145715bc,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a7715a4d463a44168bc78b9fa7a7718f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:56:47Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:56:47Z,use_quota=True,user_id='db307623f73f4532a96342f551ba011c',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:56:47.263337 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (242d054f-ccc6-4cf9-9eed-2d4a16266b55) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:56:47.291757 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (242d054f-ccc6-4cf9-9eed-2d4a16266b55) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:56:47.298976 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1a2844e-dbe5-490e-ae13-5397b42e7492) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:56:47.305050 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (a1a2844e-dbe5-490e-ae13-5397b42e7492) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '0adb6e45-9611-454a-af2d-365a145715bc', 'volume_name': 'volume-0adb6e45-9611-454a-af2d-365a145715bc', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:56:47.310553 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c1c191c5-6365-47e0-901d-02c4130ce5aa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:56:47.321759 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c1c191c5-6365-47e0-901d-02c4130ce5aa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:56:47.328027 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d9babf0c-474d-49f5-b601-02edcf8b5671) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:56:47.330555 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Volume 0adb6e45-9611-454a-af2d-365a145715bc: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-0adb6e45-9611-454a-af2d-365a145715bc', 'volume_size': 1}
Sep 03 20:56:47.331203 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] creating volume 'volume-0adb6e45-9611-454a-af2d-365a145715bc' {{(pid=133464) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1040}}
Sep 03 20:56:47.332439 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:47.478402 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:56:47.478790 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:56:47.482857 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (d9babf0c-474d-49f5-b601-02edcf8b5671) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-0adb6e45-9611-454a-af2d-365a145715bc', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:56:47.494157 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8331d423-b509-4283-999c-b56a97f8eac8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:56:47.514462 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Volume volume-0adb6e45-9611-454a-af2d-365a145715bc (0adb6e45-9611-454a-af2d-365a145715bc): created successfully
Sep 03 20:56:47.519064 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (8331d423-b509-4283-999c-b56a97f8eac8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:56:47.532135 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Flow 'volume_create_manager' (8d8a39eb-8487-42f7-814b-3b5c68130fa9) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:56:47.555807 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-33c29794-3e1f-49dc-80b7-c1c3eec7878a tempest-ServerRescueNegativeTestJSON-1941320848 None] Created volume successfully.
Sep 03 20:56:51.940412 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-d3a3565e-18a8-4a0d-b0b4-853a5a3a72a5 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Terminate volume connection completed successfully.
Sep 03 20:56:51.941088 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-d3a3565e-18a8-4a0d-b0b4-853a5a3a72a5 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Deleting attachment 8736066a-4b1f-45d8-a8e4-879ed457450f. {{(pid=133464) attachment_delete /opt/stack/cinder/cinder/volume/manager.py:4998}}
Sep 03 20:56:52.147683 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Acquiring lock "cinder-f6866eae-8b17-4560-881a-3622ae0208ea-delete_volume" by "delete_volume" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:56:52.168484 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Lock "cinder-f6866eae-8b17-4560-881a-3622ae0208ea-delete_volume" acquired by "delete_volume" :: waited 0.020s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:56:52.264472 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:52.468983 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] volume has no backup snaps {{(pid=133464) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1173}}
Sep 03 20:56:52.481286 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] deleting rbd volume volume-f6866eae-8b17-4560-881a-3622ae0208ea {{(pid=133464) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1325}}
Sep 03 20:56:52.518291 ncdv-master-1 cinder-volume[133464]: DEBUG oslo_service.periodic_task [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Running periodic task VolumeManager.publish_service_capabilities {{(pid=133464) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
Sep 03 20:56:52.519663 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:52.613334 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:52.760837 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-88f02c57-7fcd-44c4-b8ec-0fca7f482ec4 None None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:56:52.823995 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] volume is a clone so cleaning references {{(pid=133464) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1346}}
Sep 03 20:56:52.865794 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] deleting parent snapshot volume-f6866eae-8b17-4560-881a-3622ae0208ea.clone_snap {{(pid=133464) _delete_clone_parent_refs /opt/stack/cinder/cinder/volume/drivers/rbd.py:1239}}
Sep 03 20:56:53.985589 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Acquiring lock "cinder-7d6b5527-60b3-46e7-b592-f5108171cc0c-delete_volume" by "delete_volume" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:56:54.023784 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Lock "cinder-7d6b5527-60b3-46e7-b592-f5108171cc0c-delete_volume" acquired by "delete_volume" :: waited 0.038s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:56:54.104551 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:54.188387 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] volume has no backup snaps {{(pid=133464) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1173}}
Sep 03 20:56:54.189760 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Volume volume-7d6b5527-60b3-46e7-b592-f5108171cc0c is not a clone. {{(pid=133464) _get_clone_info /opt/stack/cinder/cinder/volume/drivers/rbd.py:1201}}
Sep 03 20:56:54.197057 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] deleting rbd volume volume-7d6b5527-60b3-46e7-b592-f5108171cc0c {{(pid=133464) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1325}}
Sep 03 20:56:54.199686 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.quota [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Created reservations ['d14d0543-ae4d-4645-a5f0-b4f927ce0d6d', 'b5aa8adb-f902-4dbb-b5dd-d9c9421d782a', 'f8cbfe38-3c72-498b-ad87-71e1a8ff100c', '6ac710d7-3d73-494e-9b5e-520abe9a4b76'] {{(pid=133464) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 03 20:56:54.289774 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:54.389354 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:54.493583 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:56:54.498033 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Deleted volume successfully.
Sep 03 20:56:54.544847 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [req-aab42a05-e28a-45f2-809f-be69028c67d0 req-30ead7e7-be56-44a4-a479-e3b5d950657f tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Lock "cinder-f6866eae-8b17-4560-881a-3622ae0208ea-delete_volume" released by "delete_volume" :: held 2.377s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:56:54.716012 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.quota [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Created reservations ['6ae06429-707e-41b3-9e8b-2b76df6c71a8', '7b23fb5c-6f1e-44a0-a003-631e809a2556', 'da5c04de-6d38-4ad9-9571-86716d488588', '1a5a5192-35e7-4ca2-b4b5-72b602e2a312'] {{(pid=133464) reserve /opt/stack/cinder/cinder/quota.py:755}}
Sep 03 20:56:54.871840 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:54.952546 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:56:55.048734 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.manager [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Notifying Schedulers of capabilities ... {{(pid=133464) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:56:55.061212 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Deleted volume successfully.
Sep 03 20:56:55.084751 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-afadd432-903b-481b-8a56-aca8e9283505 tempest-LiveAutoBlockMigrationV225Test-1007400564 None] Lock "cinder-7d6b5527-60b3-46e7-b592-f5108171cc0c-delete_volume" released by "delete_volume" :: held 1.060s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:57:12.771822 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=35,cinder:UPDATE=19,cinder:DELETE=2,cinder:INSERT=3 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:57:13.581322 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:57:13.638748 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Flow 'volume_create_manager' (d59e9901-76fc-4468-b1d5-ec143bccacef) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:57:13.658068 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (46541c1f-431c-486c-a895-ceab07945e62) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:13.695315 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (46541c1f-431c-486c-a895-ceab07945e62) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:13Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-ServerRescueNegativeTestJSON-volume-442612205',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='a7715a4d463a44168bc78b9fa7a7718f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:14Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:14Z,use_quota=True,user_id='db307623f73f4532a96342f551ba011c',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:13.703420 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (620d4ac5-b15b-456d-9c1c-30e69951d8fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:13.710220 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (620d4ac5-b15b-456d-9c1c-30e69951d8fb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:13.720492 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (21367792-2515-4d2e-99c6-f5f252e0feb2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:13.731102 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (21367792-2515-4d2e-99c6-f5f252e0feb2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'raw', 'volume_id': '2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1', 'volume_name': 'volume-2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:13.743321 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9a81868-74fc-4820-aaf3-8959599f6961) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:13.756555 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (b9a81868-74fc-4820-aaf3-8959599f6961) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:13.771394 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1d42acf2-6100-450f-a220-baf093d4ce24) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:13.775677 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Volume 2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1', 'volume_size': 1}
Sep 03 20:57:13.776763 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] creating volume 'volume-2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1' {{(pid=133464) create_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1040}}
Sep 03 20:57:13.785078 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:13.922443 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_replicated_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:915}}
Sep 03 20:57:13.923184 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] extra_specs: {'volume_backend_name': 'ceph'} {{(pid=133464) _is_multiattach_type /opt/stack/cinder/cinder/volume/drivers/rbd.py:924}}
Sep 03 20:57:13.929390 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1d42acf2-6100-450f-a220-baf093d4ce24) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'volume_name': 'volume-2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1', 'volume_size': 1}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:13.935028 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (53c5fb5b-d158-469f-a1b3-46f049c2d097) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:13.964593 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Volume volume-2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1 (2c1b4a46-e61e-4a8b-ae66-0be67cc1d2d1): created successfully
Sep 03 20:57:13.989184 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (53c5fb5b-d158-469f-a1b3-46f049c2d097) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:14.008852 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Flow 'volume_create_manager' (d59e9901-76fc-4468-b1d5-ec143bccacef) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:57:14.032450 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.manager [None req-0200979e-feaa-42f7-9ddf-ad3a11cdc36f tempest-ServerRescueNegativeTestJSON-1941320848 None] Created volume successfully.
Sep 03 20:57:22.778061 ncdv-master-1 cinder-volume[133464]: DEBUG dbcounter [-] [133464] Writing DB stats cinder:SELECT=42,cinder:UPDATE=20,cinder:INSERT=5,cinder:DELETE=2 {{(pid=133464) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:57:30.428592 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=133464) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:57:30.495212 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Flow 'volume_create_manager' (c870df16-337e-4f4d-a710-ce18200e628f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:57:30.512681 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (986df2a6-4478-4022-89b8-4022edac4d18) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:30.547709 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (986df2a6-4478-4022-89b8-4022edac4d18) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:30Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-tempest.common.compute-volume-1552930909',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=b82bd011-3b1c-4ad2-9e04-33631e236c5c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='5500274fc1834f768e8659be97e4004c',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:30Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:30Z,use_quota=True,user_id='f40d4376bec042e49ee8356e22baa812',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:30.566847 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e0da576f-19ff-4669-9432-2d5e8ab3d2f7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:30.581153 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e0da576f-19ff-4669-9432-2d5e8ab3d2f7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:30.595149 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (32786a68-2b06-4973-8ef7-eb2cc1c42719) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:30.749683 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (32786a68-2b06-4973-8ef7-eb2cc1c42719) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': 'b82bd011-3b1c-4ad2-9e04-33631e236c5c', 'volume_name': 'volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac73bac0>}' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:30.757356 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (37181861-a4bf-4d2b-962f-9b0661fce5ff) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:30.770929 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (37181861-a4bf-4d2b-962f-9b0661fce5ff) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:57:30.784514 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.manager [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e01222f1-c85c-4516-924a-775f08eed712) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133464) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:57:30.788337 ncdv-master-1 cinder-volume[133464]: INFO cinder.volume.flows.manager.create_volume [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Volume b82bd011-3b1c-4ad2-9e04-33631e236c5c: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac73bac0>}
Sep 03 20:57:30.790104 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Cloning b82bd011-3b1c-4ad2-9e04-33631e236c5c from image bb6b1dcc-96d7-4d20-8933-997caa341790  at location (None, None). {{(pid=133464) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1069}}
Sep 03 20:57:30.791043 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Acquiring lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" by "_prepare_image_cache_entry" {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:57:30.809171 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" acquired by "_prepare_image_cache_entry" :: waited 0.017s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:57:30.829698 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 57, 30, 819912)}. {{(pid=133464) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:57:30.830951 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=133464) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:57:30.831364 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Found cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=133464) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:884}}
Sep 03 20:57:30.840821 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.coordination [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" released by "_prepare_image_cache_entry" :: held 0.033s {{(pid=133464) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:57:30.843801 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Attempting to retrieve cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=133464) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:825}}
Sep 03 20:57:30.863719 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 57, 30, 854570)}. {{(pid=133464) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:57:30.864296 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.image.cache [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=133464) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:57:30.865029 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.flows.manager.create_volume [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] Creating from source image-volume 2a98a68c-b86c-4af3-b4f2-5063b211791e {{(pid=133464) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:839}}
Sep 03 20:57:30.888762 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=133464) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:31.040999 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] creating snapshot='volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c.clone_snap' {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:779}}
Sep 03 20:57:31.855232 ncdv-master-1 cinder-volume[133464]: DEBUG cinder.volume.drivers.rbd [None req-b404f541-6585-4376-b576-1d457ac4a61f tempest-ServersTestBootFromVolume-544474738 None] cloning 'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e@volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c.clone_snap' to 'volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c' {{(pid=133464) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:785}}
Sep 03 20:57:35.992827 ncdv-master-1 cinder-volume[133292]: INFO oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Child 133464 killed by signal 11
Sep 03 20:57:35.998105 ncdv-master-1 cinder-volume[133292]: DEBUG oslo_service.service [None req-c7de7f62-6d20-43bd-a8f8-09a327fd16fd None None] Started child 150699 {{(pid=133292) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}}
Sep 03 20:57:36.010942 ncdv-master-1 cinder-volume[150699]: INFO cinder.service [-] Starting cinder-volume node (version 20.1.0)
Sep 03 20:57:36.106694 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Starting volume driver RBDDriver (1.2.0)
Sep 03 20:57:36.108279 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:36.208068 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:36.266581 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:36.328683 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.driver [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Driver hasn't implemented _init_vendor_properties()
Sep 03 20:57:36.329307 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.driver [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Initialized capabilities list: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.46, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:35844ec5-7783-41e2-937f-44a580242021:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'properties': {'thin_provisioning': {'title': 'Thin Provisioning', 'description': 'Sets thin provisioning.', 'type': 'boolean'}, 'compression': {'title': 'Compression', 'description': 'Enables compression.', 'type': 'boolean'}, 'qos': {'title': 'QoS', 'description': 'Enables QoS.', 'type': 'boolean'}, 'replication_enabled': {'title': 'Replication', 'description': 'Enables replication.', 'type': 'boolean'}}}. {{(pid=150699) init_capabilities /opt/stack/cinder/cinder/volume/driver.py:812}}
Sep 03 20:57:36.353148 ncdv-master-1 cinder-volume[150699]: DEBUG oslo_db.sqlalchemy.engines [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=150699) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}}
Sep 03 20:57:36.356771 ncdv-master-1 cinder-volume[150699]: DEBUG dbcounter [-] [150699] Writer thread running {{(pid=150699) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}}
Sep 03 20:57:36.409330 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.db.sqlalchemy.api [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Building query based on filter {{(pid=150699) _process_snaps_filters /opt/stack/cinder/cinder/db/sqlalchemy/api.py:3885}}
Sep 03 20:57:36.448918 ncdv-master-1 cinder-volume[150699]: INFO cinder.keymgr.migration [None req-960790b4-e87d-4536-8fbb-b1222280138f None None] Not migrating encryption keys because the ConfKeyManager is still in use.
Sep 03 20:57:36.453850 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:36.540140 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:36.601718 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Notifying Schedulers of capabilities ... {{(pid=150699) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}
Sep 03 20:57:36.658424 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Driver initialization completed successfully.
Sep 03 20:57:36.659174 ncdv-master-1 cinder-volume[150699]: INFO cinder.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Initiating service 3 cleanup
Sep 03 20:57:36.753449 ncdv-master-1 cinder-volume[150699]: INFO cinder.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Cleaning Volume with id b82bd011-3b1c-4ad2-9e04-33631e236c5c and status creating
Sep 03 20:57:36.776165 ncdv-master-1 cinder-volume[150699]: INFO cinder.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Service 3 cleanup completed.
Sep 03 20:57:36.778292 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.service [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Creating RPC server for service cinder-volume {{(pid=150699) start /opt/stack/cinder/cinder/service.py:238}}
Sep 03 20:57:36.798458 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.service [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=150699) start /opt/stack/cinder/cinder/service.py:244}}
Sep 03 20:57:36.852823 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)
Sep 03 20:57:36.855332 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:36.928661 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:37.023007 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.manager [None req-b9b7340c-8c7f-4b94-a715-e3c55f5d632f None None] Driver post RPC initialization completed successfully.
Sep 03 20:57:37.114235 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] Acquiring lock "cinder-b82bd011-3b1c-4ad2-9e04-33631e236c5c-delete_volume" by "delete_volume" {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:57:37.157826 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] Lock "cinder-b82bd011-3b1c-4ad2-9e04-33631e236c5c-delete_volume" acquired by "delete_volume" :: waited 0.044s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:57:37.296030 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:57:37.431800 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] volume has no backup snaps {{(pid=150699) _delete_backup_snaps /opt/stack/cinder/cinder/volume/drivers/rbd.py:1173}}
Sep 03 20:57:37.443108 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] deleting rbd volume volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c {{(pid=150699) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1325}}
Sep 03 20:57:37.811707 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] volume is a clone so cleaning references {{(pid=150699) delete_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:1346}}
Sep 03 20:57:37.889949 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] deleting parent snapshot volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c.clone_snap {{(pid=150699) _delete_clone_parent_refs /opt/stack/cinder/cinder/volume/drivers/rbd.py:1239}}
Sep 03 20:58:01.076229 ncdv-master-1 cinder-volume[150699]: DEBUG dbcounter [-] [150699] Writing DB stats cinder:SELECT=10,cinder:DELETE=1,cinder:UPDATE=1 {{(pid=150699) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:58:01.081534 ncdv-master-1 cinder-volume[150699]: DEBUG dbcounter [-] [150699] Writing DB stats cinder:SELECT=15,cinder:UPDATE=2 {{(pid=150699) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}
Sep 03 20:58:01.160275 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] Lock "cinder-b82bd011-3b1c-4ad2-9e04-33631e236c5c-delete_volume" released by "delete_volume" :: held 24.002s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server [None req-e67a7e2e-5d5e-4b9f-b1e2-d455dbda5141 tempest-ServersTestBootFromVolume-544474738 None] Exception during message handling: rbd.ReadOnlyImage: [errno 30] RBD read-only image (error removing snapshot b'volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 203, in wrapper
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     skip_clean = func(self, context, volume, *args, **kwargs)
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/coordination.py", line 200, in _synchronized
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return f(*a, **k)
Sep 03 20:58:01.203079 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1042, in delete_volume
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     self._clear_db(volume, new_status)
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise self.value
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 1026, in delete_volume
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     self.driver.delete_volume(volume)
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 1348, in delete_volume
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     self._delete_clone_parent_refs(client, parent, parent_snap)
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 1241, in _delete_clone_parent_refs
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     parent_rbd.remove_snap(parent_snap)
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3551, in rbd.Image.remove_snap
Sep 03 20:58:01.203957 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error removing snapshot b'volume-b82bd011-3b1c-4ad2-9e04-33631e236c5c.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:01.205610 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:01.219574 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume reschedule parameters: True retry: {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']} {{(pid=150699) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:58:01.364703 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Flow 'volume_create_manager' (775815c0-fdab-4931-8a0c-ad0323ba4944) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:58:01.373612 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fba98a2b-4df9-457d-a655-11691949652d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.397571 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fba98a2b-4df9-457d-a655-11691949652d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:46Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:01.404272 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7030f387-aba5-4254-b6f2-09fe789a9c43) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.410092 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7030f387-aba5-4254-b6f2-09fe789a9c43) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:01.415719 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b787470d-9d32-4e76-a77d-27297c3da380) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.577731 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b787470d-9d32-4e76-a77d-27297c3da380) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac0d8a90>}' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:01.587180 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c0dbe89d-f22d-4dad-b738-707fa55d6e72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.592674 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c0dbe89d-f22d-4dad-b738-707fa55d6e72) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:01.599429 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (53fefd5e-de94-495a-a630-7d479fd75210) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.601081 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac0d8a90>}
Sep 03 20:58:01.602101 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Cloning 5a0d76b1-45e2-46e6-9baa-ebcef95923d4 from image bb6b1dcc-96d7-4d20-8933-997caa341790  at location (None, None). {{(pid=150699) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1069}}
Sep 03 20:58:01.603116 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Acquiring lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" by "_prepare_image_cache_entry" {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:58:01.615809 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" acquired by "_prepare_image_cache_entry" :: waited 0.012s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:58:01.646490 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 58, 1, 631350)}. {{(pid=150699) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:58:01.647783 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=150699) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:58:01.648479 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=150699) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:884}}
Sep 03 20:58:01.656791 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" released by "_prepare_image_cache_entry" :: held 0.041s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:58:01.658688 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Attempting to retrieve cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=150699) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:825}}
Sep 03 20:58:01.669411 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 58, 1, 666082)}. {{(pid=150699) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:58:01.670929 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=150699) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:58:01.672325 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Creating from source image-volume 2a98a68c-b86c-4af3-b4f2-5063b211791e {{(pid=150699) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:839}}
Sep 03 20:58:01.706209 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:58:01.818560 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] creating snapshot='volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' {{(pid=150699) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:779}}
Sep 03 20:58:01.868989 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (53fefd5e-de94-495a-a630-7d479fd75210) transitioned into state 'FAILURE' from state 'RUNNING'
Sep 03 20:58:01.868989 ncdv-master-1 cinder-volume[150699]: 5 predecessors (most recent first):
Sep 03 20:58:01.868989 ncdv-master-1 cinder-volume[150699]:   Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac2f4d30>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:46Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}, 'provides': None}
Sep 03 20:58:01.869313 ncdv-master-1 cinder-volume[150699]:   |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac2f4d30>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:46Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc), 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=bb6b1dcc-96d7-4d20-8933-997caa341790,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(5a0d76b1-45e2-46e6-9baa-ebcef95923d4),volume_id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,volume_properties=VolumeProperties,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4ac0d8a90>}}
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]:      |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'retry': {'num_attempts': 1, 'backends': ['ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph']}, 'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'snapshot_id': None, 'source_volid': None, 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume': {'id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', '_name_id': None, 'ec2_id': None, 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'volume_type': {'id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'name': 'ceph', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-03T20:39:42.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2022-09-03T20:57:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'name_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'name': 'ceph', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-03T20:39:42.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'qos_specs': None, 'reservations': ['d3d4bdb2-4fa4-47b2-88b4-3aab9ce5760f', '0953e8b8-96ad-4c19-851c-6555bc49bec0', 'efcbc943-f9b6-492b-bc69-53f2bb5061ef', 'a1cf57ac-07ed-43c9-88d0-f61179aa9d7f'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'qos_specs': None, 'reservations': ['d3d4bdb2-4fa4-47b2-88b4-3aab9ce5760f', '0953e8b8-96ad-4c19-851c-6555bc49bec0', 'efcbc943-f9b6-492b-bc69-53f2bb5061ef', 'a1cf57ac-07ed-43c9-88d0-f61179aa9d7f'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2022-09-03T20:39:42Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=d5f67b46-da42-4da6-ba4d-62966baf81fc,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2022-09-03T20:39:42Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=d5f67b46-da42-4da6-ba4d-62966baf81fc,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'metadata': {}, 'qos_specs': None}, 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=bb6b1dcc-96d7-4d20-8933-997caa341790,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(5a0d76b1-45e2-46e6-9baa-ebcef95923d4),volume_id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,volume_properties=VolumeProperties,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc)), 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:46Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc), 'context': <cinder.context.RequestContext object at 0x7fb4ac2f4d30>}, 'provides': None}
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]:         |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac2f4d30>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:46Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:57:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:57:46Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]:            |__Flow 'volume_create_manager': rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 782, in create_cloned_volume
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     src_volume.create_snap(clone_snap)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     result = task.execute(**arguments)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1235, in execute
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_image(context,
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return r(f, *args, **kwargs)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return fut.result()
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return self.__get_result()
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     raise self._exception
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     result = fn(*args, **kwargs)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1123, in _create_from_image
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_image_cache_or_download(
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 942, in _create_from_image_cache_or_download
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update, cloned = self._create_from_image_cache(
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 841, in _create_from_image_cache
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_source_volume(
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 657, in _create_from_source_volume
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 793, in create_cloned_volume
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     src_volume.unprotect_snap(clone_snap)
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:01.869461 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:01.926346 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (53fefd5e-de94-495a-a630-7d479fd75210) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.932285 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (53fefd5e-de94-495a-a630-7d479fd75210) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:01.944257 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c0dbe89d-f22d-4dad-b738-707fa55d6e72) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.949989 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c0dbe89d-f22d-4dad-b738-707fa55d6e72) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:01.970196 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b787470d-9d32-4e76-a77d-27297c3da380) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.976560 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (b787470d-9d32-4e76-a77d-27297c3da380) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:01.986961 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7030f387-aba5-4254-b6f2-09fe789a9c43) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:01.988576 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Updating volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4 with {'scheduled_at': datetime.datetime(2022, 9, 3, 20, 58, 1, 988147), 'host': None}. {{(pid=150699) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:138}}
Sep 03 20:58:02.016761 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: re-scheduling SchedulerAPI.create_volume attempt 1 due to [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap') {{(pid=150699) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:161}}
Sep 03 20:58:02.069574 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: re-scheduled {{(pid=150699) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:178}}
Sep 03 20:58:02.077245 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:58:02.206998 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] volume volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4 no longer exists in backend
Sep 03 20:58:02.214103 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (7030f387-aba5-4254-b6f2-09fe789a9c43) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True'
Sep 03 20:58:02.229342 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fba98a2b-4df9-457d-a655-11691949652d) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.233742 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (fba98a2b-4df9-457d-a655-11691949652d) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:02.239449 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Flow 'volume_create_manager' (775815c0-fdab-4931-8a0c-ad0323ba4944) transitioned into state 'REVERTED' from state 'RUNNING'
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Exception during message handling: rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 782, in create_cloned_volume
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     src_volume.create_snap(clone_snap)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     _run_flow()
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     flow_engine.run()
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     failures[0].reraise()
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 350, in reraise
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise value
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1235, in execute
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image(context,
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return fut.result()
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self.__get_result()
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise self._exception
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1123, in _create_from_image
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image_cache_or_download(
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 942, in _create_from_image_cache_or_download
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update, cloned = self._create_from_image_cache(
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 841, in _create_from_image_cache
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_source_volume(
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 657, in _create_from_source_volume
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 793, in create_cloned_volume
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     src_volume.unprotect_snap(clone_snap)
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:02.260730 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:02.311605 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume reschedule parameters: True retry: {'num_attempts': 2, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']} {{(pid=150699) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:58:02.347639 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Flow 'volume_create_manager' (05b3d7b7-24db-4772-b0e8-697c33e9dfad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:58:02.355820 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3cced261-6a71-4630-bffd-6837c9a3d40c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.385209 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3cced261-6a71-4630-bffd-6837c9a3d40c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:02Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:02.398198 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e48c5b85-b8df-4923-87aa-cd75da348ff0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.404472 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e48c5b85-b8df-4923-87aa-cd75da348ff0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:02.412557 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (68161bd3-bd38-412f-9aea-c9087328315f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.526755 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (68161bd3-bd38-412f-9aea-c9087328315f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4abfb3eb0>}' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:02.539793 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (520c473f-b57f-478d-a75a-197e6bfcf417) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.551213 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (520c473f-b57f-478d-a75a-197e6bfcf417) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:02.557220 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (55ed81ef-d9eb-4c5b-a494-25260478729c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.559474 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4abfb3eb0>}
Sep 03 20:58:02.560518 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Cloning 5a0d76b1-45e2-46e6-9baa-ebcef95923d4 from image bb6b1dcc-96d7-4d20-8933-997caa341790  at location (None, None). {{(pid=150699) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1069}}
Sep 03 20:58:02.561599 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Acquiring lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" by "_prepare_image_cache_entry" {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:58:02.580050 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" acquired by "_prepare_image_cache_entry" :: waited 0.016s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:58:02.609390 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 58, 2, 597341)}. {{(pid=150699) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:58:02.609814 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=150699) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:58:02.611380 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=150699) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:884}}
Sep 03 20:58:02.622547 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" released by "_prepare_image_cache_entry" :: held 0.044s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:58:02.623228 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Attempting to retrieve cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=150699) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:825}}
Sep 03 20:58:02.644026 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 58, 2, 632000)}. {{(pid=150699) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:58:02.646048 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=150699) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:58:02.647899 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Creating from source image-volume 2a98a68c-b86c-4af3-b4f2-5063b211791e {{(pid=150699) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:839}}
Sep 03 20:58:02.678709 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:58:02.797358 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] creating snapshot='volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' {{(pid=150699) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:779}}
Sep 03 20:58:02.829742 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (55ed81ef-d9eb-4c5b-a494-25260478729c) transitioned into state 'FAILURE' from state 'RUNNING'
Sep 03 20:58:02.829742 ncdv-master-1 cinder-volume[150699]: 5 predecessors (most recent first):
Sep 03 20:58:02.829742 ncdv-master-1 cinder-volume[150699]:   Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac0791c0>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:02Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}, 'provides': None}
Sep 03 20:58:02.829742 ncdv-master-1 cinder-volume[150699]:   |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac0791c0>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:02Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc), 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=bb6b1dcc-96d7-4d20-8933-997caa341790,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(5a0d76b1-45e2-46e6-9baa-ebcef95923d4),volume_id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,volume_properties=VolumeProperties,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4abfb3eb0>}}
Sep 03 20:58:02.829742 ncdv-master-1 cinder-volume[150699]:      |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'snapshot_id': None, 'source_volid': None, 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume': {'id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', '_name_id': None, 'ec2_id': None, 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'name': 'ceph', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-03T20:39:42.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2022-09-03T20:57:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'name_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'name': 'ceph', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-03T20:39:42.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'qos_specs': None, 'reservations': ['d3d4bdb2-4fa4-47b2-88b4-3aab9ce5760f', '0953e8b8-96ad-4c19-851c-6555bc49bec0', 'efcbc943-f9b6-492b-bc69-53f2bb5061ef', 'a1cf57ac-07ed-43c9-88d0-f61179aa9d7f'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'qos_specs': None, 'reservations': ['d3d4bdb2-4fa4-47b2-88b4-3aab9ce5760f', '0953e8b8-96ad-4c19-851c-6555bc49bec0', 'efcbc943-f9b6-492b-bc69-53f2bb5061ef', 'a1cf57ac-07ed-43c9-88d0-f61179aa9d7f'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2022-09-03T20:39:42Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=d5f67b46-da42-4da6-ba4d-62966baf81fc,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2022-09-03T20:39:42Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=d5f67b46-da42-4da6-ba4d-62966baf81fc,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 2, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']}}, 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=bb6b1dcc-96d7-4d20-8933-997caa341790,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(5a0d76b1-45e2-46e6-9baa-ebcef95923d4),volume_id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,volume_properties=VolumeProperties,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc)), 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:02Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc), 'context': <cinder.context.RequestContext object at 0x7fb4ac0791c0>}, 'provides': None}
Sep 03 20:58:02.829742 ncdv-master-1 cinder-volume[150699]:         |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac0791c0>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:02Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:02Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:02Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]:            |__Flow 'volume_create_manager': rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 782, in create_cloned_volume
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     src_volume.create_snap(clone_snap)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     result = task.execute(**arguments)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1235, in execute
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_image(context,
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return r(f, *args, **kwargs)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return fut.result()
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return self.__get_result()
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     raise self._exception
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     result = fn(*args, **kwargs)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1123, in _create_from_image
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_image_cache_or_download(
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 942, in _create_from_image_cache_or_download
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update, cloned = self._create_from_image_cache(
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 841, in _create_from_image_cache
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_source_volume(
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 657, in _create_from_source_volume
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 793, in create_cloned_volume
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     src_volume.unprotect_snap(clone_snap)
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:02.830415 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:02.854413 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (55ed81ef-d9eb-4c5b-a494-25260478729c) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.869340 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (55ed81ef-d9eb-4c5b-a494-25260478729c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:02.874810 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (520c473f-b57f-478d-a75a-197e6bfcf417) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.888992 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (520c473f-b57f-478d-a75a-197e6bfcf417) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:02.906610 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (68161bd3-bd38-412f-9aea-c9087328315f) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.913550 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (68161bd3-bd38-412f-9aea-c9087328315f) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:02.924532 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e48c5b85-b8df-4923-87aa-cd75da348ff0) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:02.924532 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Updating volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4 with {'scheduled_at': datetime.datetime(2022, 9, 3, 20, 58, 2, 922597), 'host': None}. {{(pid=150699) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:138}}
Sep 03 20:58:02.934087 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: re-scheduling SchedulerAPI.create_volume attempt 2 due to [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap') {{(pid=150699) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:161}}
Sep 03 20:58:02.998094 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: re-scheduled {{(pid=150699) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:178}}
Sep 03 20:58:02.998878 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:58:03.085152 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] volume volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4 no longer exists in backend
Sep 03 20:58:03.097414 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (e48c5b85-b8df-4923-87aa-cd75da348ff0) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True'
Sep 03 20:58:03.107512 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3cced261-6a71-4630-bffd-6837c9a3d40c) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.114225 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (3cced261-6a71-4630-bffd-6837c9a3d40c) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:03.138867 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Flow 'volume_create_manager' (05b3d7b7-24db-4772-b0e8-697c33e9dfad) transitioned into state 'REVERTED' from state 'RUNNING'
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Exception during message handling: rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 782, in create_cloned_volume
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     src_volume.create_snap(clone_snap)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     _run_flow()
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     flow_engine.run()
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     failures[0].reraise()
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 350, in reraise
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise value
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1235, in execute
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image(context,
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return fut.result()
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self.__get_result()
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise self._exception
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1123, in _create_from_image
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image_cache_or_download(
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 942, in _create_from_image_cache_or_download
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update, cloned = self._create_from_image_cache(
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 841, in _create_from_image_cache
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_source_volume(
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 657, in _create_from_source_volume
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 793, in create_cloned_volume
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     src_volume.unprotect_snap(clone_snap)
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:03.140956 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:03.263213 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume reschedule parameters: True retry: {'num_attempts': 3, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']} {{(pid=150699) get_flow /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1366}}
Sep 03 20:58:03.318956 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Flow 'volume_create_manager' (deaf9ecf-348d-4095-8739-66652b231465) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}}
Sep 03 20:58:03.335919 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (32825361-4e11-40c8-91df-de6073735f39) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.378368 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (32825361-4e11-40c8-91df-de6073735f39) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:03Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:03.387761 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (710e78e7-2cdc-4935-b170-0346023b5b19) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.395429 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (710e78e7-2cdc-4935-b170-0346023b5b19) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:03.400883 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (849b5b71-dc58-4b67-8996-ef6a711b00c2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.506341 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (849b5b71-dc58-4b67-8996-ef6a711b00c2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': 'creating', 'type': 'image', 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4abf1f730>}' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:03.511727 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc49e993-4da8-445a-88c5-c3a612a23188) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.515593 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc49e993-4da8-445a-88c5-c3a612a23188) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}}
Sep 03 20:58:03.521534 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0cc4be3c-7ff5-4af9-b46e-540403ffd300) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.522806 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4abf1f730>}
Sep 03 20:58:03.523556 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Cloning 5a0d76b1-45e2-46e6-9baa-ebcef95923d4 from image bb6b1dcc-96d7-4d20-8933-997caa341790  at location (None, None). {{(pid=150699) _create_from_image /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:1069}}
Sep 03 20:58:03.524248 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Acquiring lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" by "_prepare_image_cache_entry" {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:192}}
Sep 03 20:58:03.534315 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" acquired by "_prepare_image_cache_entry" :: waited 0.010s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:196}}
Sep 03 20:58:03.559198 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 58, 3, 539212)}. {{(pid=150699) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:58:03.559512 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=150699) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:58:03.559902 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=150699) _prepare_image_cache_entry /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:884}}
Sep 03 20:58:03.565737 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.coordination [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Lock "cinder-bb6b1dcc-96d7-4d20-8933-997caa341790" released by "_prepare_image_cache_entry" :: held 0.031s {{(pid=150699) _synchronized /opt/stack/cinder/cinder/coordination.py:207}}
Sep 03 20:58:03.568092 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Attempting to retrieve cache entry for image = bb6b1dcc-96d7-4d20-8933-997caa341790 on host ncdv-master-1@ceph#ceph. {{(pid=150699) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:825}}
Sep 03 20:58:03.575805 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Found image-volume cache entry: {'id': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'volume_id': '2a98a68c-b86c-4af3-b4f2-5063b211791e', 'host': 'ncdv-master-1@ceph#ceph', 'size': 1, 'image_updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15), 'last_used': datetime.datetime(2022, 9, 3, 20, 58, 3, 572932)}. {{(pid=150699) get_entry /opt/stack/cinder/cinder/image/cache.py:76}}
Sep 03 20:58:03.576236 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.image.cache [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Image-volume cache entry image_update_at = 2022-09-03 20:40:15+00:00, requested image updated_at = 2022-09-03 20:40:15+00:00. {{(pid=150699) _should_update_entry /opt/stack/cinder/cinder/image/cache.py:259}}
Sep 03 20:58:03.576691 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Creating from source image-volume 2a98a68c-b86c-4af3-b4f2-5063b211791e {{(pid=150699) _create_from_image_cache /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:839}}
Sep 03 20:58:03.600583 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:58:03.698053 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] creating snapshot='volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' {{(pid=150699) create_cloned_volume /opt/stack/cinder/cinder/volume/drivers/rbd.py:779}}
Sep 03 20:58:03.729409 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0cc4be3c-7ff5-4af9-b46e-540403ffd300) transitioned into state 'FAILURE' from state 'RUNNING'
Sep 03 20:58:03.729409 ncdv-master-1 cinder-volume[150699]: 5 predecessors (most recent first):
Sep 03 20:58:03.729409 ncdv-master-1 cinder-volume[150699]:   Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac0013d0>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:03Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}, 'provides': None}
Sep 03 20:58:03.729409 ncdv-master-1 cinder-volume[150699]:   |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac0013d0>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:03Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc), 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=bb6b1dcc-96d7-4d20-8933-997caa341790,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(5a0d76b1-45e2-46e6-9baa-ebcef95923d4),volume_id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,volume_properties=VolumeProperties,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc))}, 'provides': {'status': 'creating', 'type': 'image', 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_size': 1, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'image_location': (None, None), 'image_meta': {'name': 'ubuntu-16', 'disk_format': 'vhdx', 'container_format': 'bare', 'visibility': 'public', 'size': 645922816, 'virtual_size': 734003200, 'status': 'active', 'checksum': '318f990048c8a194829e52141534610d', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '53641d46504c444d9350b7e021f6a4a4', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '1aaa7a04b14d2fb410b8803fefeb767485c32e2da475b52a8839eec757ebef67d8611e4f699200324e39744ee6949dd3c2a4f6ecccb0c2480706ec5cb0fc5a53', 'id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'created_at': datetime.datetime(2022, 9, 3, 20, 39, 58, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2022, 9, 3, 20, 40, 15, tzinfo=datetime.timezone.utc), 'tags': [], 'file': '/v2/images/bb6b1dcc-96d7-4d20-8933-997caa341790/file', 'properties': {'hw_rng_model': 'virtio', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/ubuntu-16', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb4abf1f730>}}
Sep 03 20:58:03.729409 ncdv-master-1 cinder-volume[150699]:      |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'filter_properties': {'request_spec': {'consistencygroup_id': None, 'group_id': None, 'cgsnapshot_id': None, 'image_id': 'bb6b1dcc-96d7-4d20-8933-997caa341790', 'snapshot_id': None, 'source_volid': None, 'volume_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume': {'id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', '_name_id': None, 'ec2_id': None, 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'use_quota': True, 'snapshot_id': None, 'cluster_name': None, 'host': None, 'size': 1, 'availability_zone': 'nova', 'status': 'creating', 'attach_status': 'detached', 'migration_status': None, 'scheduled_at': None, 'launched_at': None, 'terminated_at': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'display_description': None, 'provider_id': None, 'provider_location': None, 'provider_auth': None, 'provider_geometry': None, 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'source_volid': None, 'encryption_key_id': None, 'consistencygroup_id': None, 'group_id': None, 'deleted': False, 'bootable': False, 'multiattach': False, 'replication_status': None, 'replication_extended_status': None, 'replication_driver_data': None, 'previous_status': None, 'metadata': {}, 'admin_metadata': {}, 'glance_metadata': {}, 'volume_type': {'id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'name': 'ceph', 'description': None, 'is_public': True, 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-03T20:39:42.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_attachment': [], 'service_uuid': None, 'shared_targets': True, 'created_at': '2022-09-03T20:57:45.000000', 'updated_at': None, 'deleted_at': None, 'name': 'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'name_id': '5a0d76b1-45e2-46e6-9baa-ebcef95923d4', 'volume_metadata': [], 'volume_admin_metadata': [], 'volume_glance_metadata': []}, 'volume_type': {'id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc', 'name': 'ceph', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'ceph'}, 'qos_specs_id': None, 'created_at': '2022-09-03T20:39:42.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False}, 'volume_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'qos_specs': None, 'reservations': ['d3d4bdb2-4fa4-47b2-88b4-3aab9ce5760f', '0953e8b8-96ad-4c19-851c-6555bc49bec0', 'efcbc943-f9b6-492b-bc69-53f2bb5061ef', 'a1cf57ac-07ed-43c9-88d0-f61179aa9d7f'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc'}, 'backup_id': None, 'availability_zones': ['nova'], 'resource_properties': {'attach_status': 'detached', 'availability_zone': 'nova', 'cgsnapshot_id': None, 'consistencygroup_id': None, 'group_id': None, 'display_description': None, 'display_name': 'tempest-TaggedBootDevicesTest_v242-volume-427066127', 'encryption_key_id': None, 'metadata': {}, 'multiattach': False, 'project_id': '090fc6c9de624a209a90631c0d22d48f', 'qos_specs': None, 'reservations': ['d3d4bdb2-4fa4-47b2-88b4-3aab9ce5760f', '0953e8b8-96ad-4c19-851c-6555bc49bec0', 'efcbc943-f9b6-492b-bc69-53f2bb5061ef', 'a1cf57ac-07ed-43c9-88d0-f61179aa9d7f'], 'size': 1, 'snapshot_id': None, 'source_volid': None, 'status': 'creating', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'volume_type_id': 'd5f67b46-da42-4da6-ba4d-62966baf81fc'}}, 'config_options': {}, 'volume_type': VolumeType(created_at=2022-09-03T20:39:42Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=d5f67b46-da42-4da6-ba4d-62966baf81fc,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'resource_type': VolumeType(created_at=2022-09-03T20:39:42Z,deleted=False,deleted_at=None,description=None,extra_specs={volume_backend_name='ceph'},id=d5f67b46-da42-4da6-ba4d-62966baf81fc,is_public=True,name='ceph',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), 'size': 1, 'availability_zone': 'nova', 'user_id': '7838d64ab35e4608b668e9f65d594d6e', 'metadata': {}, 'qos_specs': None, 'retry': {'num_attempts': 3, 'backends': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph'], 'hosts': ['ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph', 'ncdv-master-1@ceph#ceph']}}, 'request_spec': RequestSpec(CG_backend=<?>,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=bb6b1dcc-96d7-4d20-8933-997caa341790,operation=<?>,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(5a0d76b1-45e2-46e6-9baa-ebcef95923d4),volume_id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,volume_properties=VolumeProperties,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc)), 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:03Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc), 'context': <cinder.context.RequestContext object at 0x7fb4ac0013d0>}, 'provides': None}
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]:         |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7fb4ac0013d0>, 'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:03Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}, 'provides': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2022-09-03T20:57:45Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-TaggedBootDevicesTest_v242-volume-427066127',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ncdv-master-1@ceph#ceph',id=5a0d76b1-45e2-46e6-9baa-ebcef95923d4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='090fc6c9de624a209a90631c0d22d48f',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2022-09-03T20:58:03Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2022-09-03T20:58:03Z,use_quota=True,user_id='7838d64ab35e4608b668e9f65d594d6e',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(d5f67b46-da42-4da6-ba4d-62966baf81fc),volume_type_id=d5f67b46-da42-4da6-ba4d-62966baf81fc)}
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]:            |__Flow 'volume_create_manager': rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 782, in create_cloned_volume
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     src_volume.create_snap(clone_snap)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager Traceback (most recent call last):
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     result = task.execute(**arguments)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1235, in execute
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_image(context,
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return r(f, *args, **kwargs)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     do = self.iter(retry_state=retry_state)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return fut.result()
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     return self.__get_result()
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     raise self._exception
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     result = fn(*args, **kwargs)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1123, in _create_from_image
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_image_cache_or_download(
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 942, in _create_from_image_cache_or_download
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update, cloned = self._create_from_image_cache(
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 841, in _create_from_image_cache
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self._create_from_source_volume(
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 657, in _create_from_source_volume
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 793, in create_cloned_volume
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager     src_volume.unprotect_snap(clone_snap)
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:03.730103 ncdv-master-1 cinder-volume[150699]: ERROR cinder.volume.manager 
Sep 03 20:58:03.747411 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0cc4be3c-7ff5-4af9-b46e-540403ffd300) transitioned into state 'REVERTING' from state 'FAILURE' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.755294 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (0cc4be3c-7ff5-4af9-b46e-540403ffd300) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:03.762211 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc49e993-4da8-445a-88c5-c3a612a23188) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.768213 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (bc49e993-4da8-445a-88c5-c3a612a23188) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:03.774219 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (849b5b71-dc58-4b67-8996-ef6a711b00c2) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.778799 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (849b5b71-dc58-4b67-8996-ef6a711b00c2) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:03.784581 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (710e78e7-2cdc-4935-b170-0346023b5b19) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.786104 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Updating volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4 with {'scheduled_at': datetime.datetime(2022, 9, 3, 20, 58, 3, 785677), 'host': None}. {{(pid=150699) _pre_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:138}}
Sep 03 20:58:03.802460 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: re-scheduling SchedulerAPI.create_volume attempt 3 due to [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap') {{(pid=150699) _reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:161}}
Sep 03 20:58:03.864419 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.flows.manager.create_volume [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Volume 5a0d76b1-45e2-46e6-9baa-ebcef95923d4: re-scheduled {{(pid=150699) _post_reschedule /opt/stack/cinder/cinder/volume/flows/manager/create_volume.py:178}}
Sep 03 20:58:03.865883 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] connecting to cinder@ceph (conf=/etc/ceph/ceph.conf, timeout=-1). {{(pid=150699) _do_conn /opt/stack/cinder/cinder/volume/drivers/rbd.py:523}}
Sep 03 20:58:03.953970 ncdv-master-1 cinder-volume[150699]: INFO cinder.volume.drivers.rbd [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] volume volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4 no longer exists in backend
Sep 03 20:58:03.962962 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (710e78e7-2cdc-4935-b170-0346023b5b19) transitioned into state 'REVERTED' from state 'REVERTING' with result 'True'
Sep 03 20:58:03.968115 ncdv-master-1 cinder-volume[150699]: DEBUG cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (32825361-4e11-40c8-91df-de6073735f39) transitioned into state 'REVERTING' from state 'SUCCESS' {{(pid=150699) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}}
Sep 03 20:58:03.973392 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (32825361-4e11-40c8-91df-de6073735f39) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
Sep 03 20:58:03.979250 ncdv-master-1 cinder-volume[150699]: WARNING cinder.volume.manager [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Flow 'volume_create_manager' (deaf9ecf-348d-4095-8739-66652b231465) transitioned into state 'REVERTED' from state 'RUNNING'
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server [None req-ecad3bfb-a7ff-406b-b5df-c55db3063c7b tempest-TaggedBootDevicesTest_v242-1933143257 None] Exception during message handling: rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 782, in create_cloned_volume
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     src_volume.create_snap(clone_snap)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3514, in rbd.Image.create_snap
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap' from b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e')
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server 
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return caller(func, *(extras + args), **kw)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 819, in create_volume
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     _run_flow()
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/manager.py", line 811, in _run_flow
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     flow_engine.run()
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 338, in reraise_if_any
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     failures[0].reraise()
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 350, in reraise
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise value
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1235, in execute
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image(context,
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/utils.py", line 669, in _wrapper
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 360, in iter
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return fut.result()
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     return self.__get_result()
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     raise self._exception
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1123, in _create_from_image
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_image_cache_or_download(
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 942, in _create_from_image_cache_or_download
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update, cloned = self._create_from_image_cache(
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 841, in _create_from_image_cache
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self._create_from_source_volume(
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 657, in _create_from_source_volume
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 793, in create_cloned_volume
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server     src_volume.unprotect_snap(clone_snap)
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3641, in rbd.Image.unprotect_snap
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server rbd.ImageNotFound: [errno 2] RBD image not found (error unprotecting snapshot b'volume-2a98a68c-b86c-4af3-b4f2-5063b211791e'@b'volume-5a0d76b1-45e2-46e6-9baa-ebcef95923d4.clone_snap')
Sep 03 20:58:03.982384 ncdv-master-1 cinder-volume[150699]: ERROR oslo_messaging.rpc.server