-- Logs begin at Sat 2021-11-20 20:01:12 UTC, end at Sat 2021-11-20 22:10:07 UTC. -- Nov 20 20:37:22.493638 ncdv-master-1 systemd[1]: Started Devstack devstack@c-sch.service. Nov 20 20:37:25.927843 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:37:26.223789 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.rpc [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Nov 20 20:37:26.227784 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.rpc [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Automatically selected cinder-volume RPC version 3.16 as minimum service version. Nov 20 20:37:26.234874 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:37:26.239709 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.rpc [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Nov 20 20:37:26.244777 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.rpc [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Automatically selected cinder-backup RPC version 2.2 as minimum service version. Nov 20 20:37:26.249279 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.rpc [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Nov 20 20:37:26.253615 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.rpc [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Nov 20 20:37:26.264009 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:37:26.292460 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_concurrency.lockutils [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Acquired lock "singleton_lock" {{(pid=133499) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Nov 20 20:37:26.293879 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_concurrency.lockutils [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Releasing lock "singleton_lock" {{(pid=133499) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Nov 20 20:37:26.294731 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] ******************************************************************************** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Nov 20 20:37:26.295243 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Configuration options gathered from: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Nov 20 20:37:26.295724 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Nov 20 20:37:26.296341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Nov 20 20:37:26.296846 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] ================================================================================ {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Nov 20 20:37:26.297490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allocated_capacity_weight_multiplier = -1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.298199 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allow_availability_zone_fallback = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.299125 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allow_compression_on_image_upload = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.299673 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allowed_direct_url_schemes = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.300217 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] api_rate_limit = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.301272 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] auth_strategy = keystone {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.301819 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] az_cache_duration = 3600 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.302371 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backdoor_port = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.302840 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backdoor_socket = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.303358 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_availability_zone = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.303919 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_api_class = cinder.backup.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.304599 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.305176 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_name_template = backup-%s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.305694 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_use_same_host = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.306314 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_use_temp_snapshot = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.306883 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] capacity_weight_multiplier = 1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.307410 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] chap_password = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.308087 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] chap_username = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.308680 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.309545 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_internal_tenant_project_id = e6c8e0b874174dd4abf198b0af571967 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.310130 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_internal_tenant_user_id = 3c67c5cdca734d5c91c30684265bd9ab {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.310641 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cloned_volume_same_az = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.311174 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] compression_format = gzip {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.312692 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] compute_api_class = cinder.compute.nova.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.313282 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config_dir = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.313743 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.314290 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config_source = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.315783 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.316324 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] control_exchange = openstack {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.316917 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] debug = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.317450 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_availability_zone = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.317941 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_group_type = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.318747 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.319769 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_volume_type = ceph {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.320622 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_client_cert = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.321426 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_client_cert_key = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.322243 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_data_namespace = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.324139 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_ssl_cert_path = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.325044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_ssl_cert_verify = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.325812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_use_ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.326414 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enable_force_upload = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.326966 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enable_new_services = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.327532 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enable_unsupported_driver = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.328076 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enabled_backends = ['ceph'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.328588 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enforce_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.329165 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] filter_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.329918 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_api_insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.330650 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_api_servers = ['http://10.222.0.41/image'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.331178 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_api_ssl_compression = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.332088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_ca_certificates_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.332611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_catalog_info = image:glance:publicURL {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.333175 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.335059 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.335610 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.336128 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_num_retries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.336647 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_request_timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.337208 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] goodness_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.337746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] graceful_shutdown_timeout = 5 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.338899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] group_api_class = cinder.group.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.339829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] host = ncdv-master-1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.340360 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iet_conf = /etc/iet/ietd.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.340879 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_compress_on_upload = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.341416 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_conversion_address_space_limit = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.341932 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_conversion_cpu_limit = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.342575 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.343100 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_upload_use_cinder_backend = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.343605 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_upload_use_internal_tenant = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.344110 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_volume_cache_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.344615 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_volume_cache_max_count = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.345286 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_volume_cache_max_size_gb = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.345812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] instance_format = [instance: %(uuid)s] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.346361 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.347008 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_iotype = fileio {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.347532 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_secondary_ip_addresses = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.348059 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_target_flags = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.348594 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_write_cache = on {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.349124 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_helper = tgtadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.349927 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.350548 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.351077 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.351622 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_config_append = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.352138 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.352649 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_dir = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.353277 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.353838 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_options = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.354402 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_rotate_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.354936 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_rotate_interval_type = days {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.355555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_rotation_type = none {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.356013 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.356463 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.357009 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.357526 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.357978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.358549 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_age = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.359057 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_logfile_count = 30 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.359564 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_logfile_size_mb = 200 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.360354 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_over_subscription_ratio = 20.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.360870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] message_reap_interval = 86400 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.361372 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] message_ttl = 2592000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.361871 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] monkey_patch = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.362421 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] monkey_patch_modules = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.362876 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] my_ip = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.363393 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] no_snapshot_gb_quota = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.363911 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] num_iser_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.364423 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] num_shell_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.364930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] num_volume_device_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.365553 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nvmet_ns_id = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.366209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nvmet_port_id = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.366753 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_max_limit = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.367304 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_ext_list = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.367845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.368392 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_listen = 0.0.0.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.368912 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_listen_port = 8776 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.369531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_use_ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.370202 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_workers = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.370905 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] per_volume_size_limit = -1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.371470 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] periodic_fuzzy_delay = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.371982 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] periodic_interval = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.372482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] public_endpoint = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.372997 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] publish_errors = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.373547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_backup_gigabytes = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.374087 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_backups = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.374584 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_consistencygroups = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.375155 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.375777 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_gigabytes = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.376279 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_groups = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.376794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_snapshots = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.377316 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_volumes = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.377862 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rate_limit_burst = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.378406 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rate_limit_except_level = CRITICAL {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.378933 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rate_limit_interval = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.379434 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] replication_device = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.379944 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] report_discard_supported = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.380680 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] report_interval = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.381271 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] reservation_clean_interval = 86400 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.381766 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] reservation_expire = 86400 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.382309 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] reserved_percentage = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.382821 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.383336 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.383864 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rpc_response_timeout = 120 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.384379 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] run_external_periodic_tasks = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.384930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_default_filters = ['AvailabilityZoneFilter', 'CapacityFilter', 'CapabilitiesFilter'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.386324 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_default_weighers = ['CapacityWeigher'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.386898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_driver = cinder.scheduler.filter_scheduler.FilterScheduler {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.387404 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_driver_init_wait_time = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.387974 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_host_manager = cinder.scheduler.host_manager.HostManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.388590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_json_config_location = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.389150 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.389651 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_max_attempts = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.390251 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_weight_handler = cinder.scheduler.weights.OrderedHostWeightHandler {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.391417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scst_target_driver = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.391939 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scst_target_iqn_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.392440 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_down_time = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.392963 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] snapshot_name_template = snapshot-%s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.393522 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] snapshot_same_host = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.395860 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] split_loggers = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.396417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] state_path = /opt/stack/data/cinder {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.396946 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] storage_availability_zone = nova {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.397470 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] storage_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.399646 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] syslog_log_facility = LOG_USER {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.400192 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_helper = lioadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.401707 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.402427 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.402971 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.403495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.403998 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] trace_flags = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.404516 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] transfer_api_class = cinder.transfer.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.405063 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] transport_url = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.405838 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] until_refresh = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.406517 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_chap_auth = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.407042 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_default_quota_class = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.407565 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_eventlog = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.408091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_journal = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.408606 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_json = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.409125 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.409636 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_stderr = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.411168 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_syslog = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.411846 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] verify_glance_signatures = enabled {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.412382 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_api_class = cinder.volume.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.412889 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_backend_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.413422 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_clear = zero {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.413926 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_clear_ionice = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.414489 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_clear_size = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.415019 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.415539 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_copy_bps_limit = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.416071 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_dd_blocksize = 1M {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.416915 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.417556 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_name_template = volume-%s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.419093 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_number_multiplier = -1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.419645 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_usage_audit_period = month {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.420250 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.420772 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] watch_log_file = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.421450 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.422210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.422744 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.enforce_new_defaults = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.423396 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.enforce_scope = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.423938 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.policy_default_rule = default {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.424760 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.425298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.policy_file = policy.yaml {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.425902 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.427627 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.428165 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.428679 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.429207 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.429735 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_concurrency.disable_process_locking = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.430352 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.430874 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.431527 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.432235 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.432767 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.433293 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.433812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.connection_string = messaging:// {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.434348 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.434869 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.es_doc_type = notification {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.435382 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.es_scroll_size = 10000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.435925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.es_scroll_time = 2m {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.436447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.filter_error_trace = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.437828 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.hmac_keys = SECRET_KEY {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.438549 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.sentinel_service_name = mymaster {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.439092 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.socket_timeout = 0.1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.439605 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.trace_sqlalchemy = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.440155 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_password = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.440681 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.441196 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_token = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.441831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_user = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.442612 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_admin_prefix = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.443159 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.443713 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_port = 35357 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.444225 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_protocol = https {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.445291 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_section = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.445850 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_type = password {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.446450 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_uri = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.446962 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_version = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.448344 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.cache = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.448910 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.449465 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.449983 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.delay_auth_decision = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.450555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.451064 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.http_connect_timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.451705 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.452214 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.identity_uri = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.452835 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.include_service_catalog = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.453358 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.453882 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.interface = public {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.454420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.454933 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.455447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.455981 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.456496 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.457017 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.457525 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.458192 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.458801 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.459410 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.459929 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.region_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.460976 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.461500 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.service_token_roles_required = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.462164 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.service_type = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.462831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.token_cache_time = 300 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.463957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.464503 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.backend = dogpile.cache.null {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.465011 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.backend_argument = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.465533 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.config_prefix = cache.oslo {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.466078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.debug_cache_backend = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.466600 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.467109 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.expiration_time = 600 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.467621 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_dead_retry = 300 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.468128 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.468799 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.469348 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_maxsize = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.469854 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.470429 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.470961 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_socket_timeout = 1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.471479 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.proxies = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.472121 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_allowed_ciphers = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.472635 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_cafile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.473223 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.473735 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.474310 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.474848 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] coordination.backend_url = etcd3+http://10.222.0.41:2379 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.475402 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.backend = sqlalchemy {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.475983 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.476506 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_debug = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.477026 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_parameters = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.477534 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_recycle_time = 3600 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.478088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_trace = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.478754 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_inc_retry_interval = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.479272 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_max_retries = 20 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.479791 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_max_retry_interval = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.480303 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_retry_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.480832 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.max_overflow = 50 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.481348 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.max_pool_size = 5 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.481871 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.max_retries = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.482584 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.mysql_enable_ndb = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.483236 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.483756 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.pool_timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.484284 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.retry_interval = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.484786 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.slave_connection = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.485299 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.sqlite_synchronous = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.485816 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.use_db_reconnect = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.486374 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.capabilities = [21] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.486897 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.group = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.487406 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.helper_command = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.487918 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.488436 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.489038 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.user = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.489577 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.490118 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.group = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.490617 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.helper_command = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.491276 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.491831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.492480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.user = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.493004 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.auth_section = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.493659 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.auth_type = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.494854 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.cafile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.495539 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.496071 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.collect_timing = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.496671 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.497182 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.497688 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.send_service_user_token = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.498228 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.split_loggers = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.498742 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.499364 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.backend_availability_zone = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.499895 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.chap_password = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.500413 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.chap_username = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.500930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.501430 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_client_cert = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.501941 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_client_cert_key = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.502624 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_data_namespace = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.503137 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.503762 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.504283 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_use_ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.504798 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.enable_unsupported_driver = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.505320 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.505830 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.filter_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.506368 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.goodness_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.506936 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.507460 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.507965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.508485 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.509007 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.509604 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.510239 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_iotype = fileio {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.510751 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.511340 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_target_flags = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.511870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_write_cache = on {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.512388 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_helper = tgtadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.513196 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.513872 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.514454 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.514993 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.515505 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.516022 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.num_shell_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.516530 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.517045 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.517565 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.nvmet_port_id = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.518098 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.replication_device = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.518620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.report_discard_supported = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.519132 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.reserved_percentage = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.519739 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.scst_target_driver = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.520252 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.scst_target_iqn_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.520773 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.storage_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.521305 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_helper = tgtadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.521937 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.522550 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.523200 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.523881 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.524402 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.trace_flags = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.524910 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.unique_fqdn_network = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.525417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.use_chap_auth = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.526469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.526989 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_backend_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.527514 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_clear = zero {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.528088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_clear_ionice = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.528614 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_clear_size = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.529148 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.529790 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.530347 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.531206 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.531736 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_reports.file_event_handler = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.532259 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.532777 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_reports.log_dir = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.533536 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.534347 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.534865 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.535390 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.535927 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.536437 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.536945 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.537453 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.537973 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.538536 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.539067 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.539693 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.540213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.540735 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.541325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.542567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.543094 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.543739 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.544431 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.544957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.545482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.545995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.546569 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.547091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.547620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.548268 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_version = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.548803 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.driver = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.549314 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.retry = -1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.550203 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.550711 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.transport_url = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.551238 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.auth_section = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.551773 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.auth_type = password {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.552293 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.552794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.553299 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.collect_timing = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.553909 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.554582 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.interface = public {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.555102 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.555631 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.region_name = RegionOne {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.556179 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.split_loggers = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.556688 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.557269 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.token_auth_url = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.557711 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] ******************************************************************************** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Nov 20 20:37:26.558306 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Full set of CONF: {{(pid=133499) _wait_for_exit_or_signal /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:362}} Nov 20 20:37:26.558728 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] ******************************************************************************** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Nov 20 20:37:26.559160 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] Configuration options gathered from: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Nov 20 20:37:26.559589 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Nov 20 20:37:26.560207 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Nov 20 20:37:26.560621 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] ================================================================================ {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Nov 20 20:37:26.561129 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allocated_capacity_weight_multiplier = -1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.561555 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allow_availability_zone_fallback = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.561976 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allow_compression_on_image_upload = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.562431 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] allowed_direct_url_schemes = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.562852 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] api_rate_limit = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.563270 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] auth_strategy = keystone {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.563819 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] az_cache_duration = 3600 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.564239 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backdoor_port = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.564779 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backdoor_socket = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.565198 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_availability_zone = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.565625 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_api_class = cinder.backup.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.566084 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.566515 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_name_template = backup-%s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.566942 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_use_same_host = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.567370 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backup_use_temp_snapshot = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.567802 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] capacity_weight_multiplier = 1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.568218 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] chap_password = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.568633 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] chap_username = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.569057 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.569485 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_internal_tenant_project_id = e6c8e0b874174dd4abf198b0af571967 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.569910 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_internal_tenant_user_id = 3c67c5cdca734d5c91c30684265bd9ab {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.570424 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cloned_volume_same_az = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.570856 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] compression_format = gzip {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.571273 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] compute_api_class = cinder.compute.nova.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.571705 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config_dir = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.572127 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.572651 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] config_source = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.573108 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.573540 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] control_exchange = openstack {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.574115 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] debug = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.574666 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_availability_zone = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.575092 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_group_type = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.575601 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.576036 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] default_volume_type = ceph {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.576474 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_client_cert = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.576892 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_client_cert_key = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.577316 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_data_namespace = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.577731 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_ssl_cert_path = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.578188 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_ssl_cert_verify = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.578607 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] driver_use_ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.579049 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enable_force_upload = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.579466 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enable_new_services = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.579888 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enable_unsupported_driver = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.580419 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enabled_backends = ['ceph'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.580839 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] enforce_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.581277 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] filter_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.581698 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_api_insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.582149 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_api_servers = ['http://10.222.0.41/image'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.582571 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_api_ssl_compression = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.582990 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_ca_certificates_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.583422 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_catalog_info = image:glance:publicURL {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.583852 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.584394 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.584952 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.585375 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_num_retries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.585804 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] glance_request_timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.586269 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] goodness_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.586690 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] graceful_shutdown_timeout = 5 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.587108 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] group_api_class = cinder.group.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.587527 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] host = ncdv-master-1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.588000 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iet_conf = /etc/iet/ietd.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.589081 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_compress_on_upload = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.589681 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_conversion_address_space_limit = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.590131 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_conversion_cpu_limit = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.590650 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.591091 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_upload_use_cinder_backend = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.591674 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_upload_use_internal_tenant = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.592106 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_volume_cache_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.592526 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_volume_cache_max_count = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.592951 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] image_volume_cache_max_size_gb = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.593371 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] instance_format = [instance: %(uuid)s] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.593788 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.594421 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_iotype = fileio {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.594971 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_secondary_ip_addresses = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.595439 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_target_flags = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.595856 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iscsi_write_cache = on {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.596284 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_helper = tgtadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.596712 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.597328 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.597748 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.598219 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_config_append = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.599145 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.599587 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_dir = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.600005 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.600531 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_options = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.600952 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_rotate_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.601380 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_rotate_interval_type = days {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.601796 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] log_rotation_type = none {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.602257 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.602690 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.603109 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.603537 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.604723 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.605395 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_age = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.605833 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_logfile_count = 30 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.606286 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_logfile_size_mb = 200 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.606705 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] max_over_subscription_ratio = 20.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.607139 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] message_reap_interval = 86400 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.607626 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] message_ttl = 2592000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.608047 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] monkey_patch = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.608473 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] monkey_patch_modules = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.608889 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] my_ip = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.609314 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] no_snapshot_gb_quota = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.609741 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] num_iser_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.610193 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] num_shell_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.610730 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] num_volume_device_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.611156 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nvmet_ns_id = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.611581 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nvmet_port_id = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.611998 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_max_limit = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.612430 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_ext_list = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.612858 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.613276 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_listen = 0.0.0.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.613693 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_listen_port = 8776 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.614128 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_use_ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.614560 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] osapi_volume_workers = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.614977 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] per_volume_size_limit = -1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.615609 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] periodic_fuzzy_delay = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.616040 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] periodic_interval = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.616459 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] public_endpoint = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.616900 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] publish_errors = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.617319 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_backup_gigabytes = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.617738 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_backups = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.618271 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_consistencygroups = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.618693 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.619125 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_gigabytes = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.619597 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_groups = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.620075 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_snapshots = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.620567 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] quota_volumes = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.621081 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rate_limit_burst = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.621516 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rate_limit_except_level = CRITICAL {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.621949 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rate_limit_interval = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.622388 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] replication_device = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.622808 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] report_discard_supported = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.623225 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] report_interval = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.623640 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] reservation_clean_interval = 86400 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.624059 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] reservation_expire = 86400 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.624492 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] reserved_percentage = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.624911 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.625538 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.625975 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] rpc_response_timeout = 120 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.626427 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] run_external_periodic_tasks = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.626875 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_default_filters = ['AvailabilityZoneFilter', 'CapacityFilter', 'CapabilitiesFilter'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.627307 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_default_weighers = ['CapacityWeigher'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.627736 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_driver = cinder.scheduler.filter_scheduler.FilterScheduler {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.628155 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_driver_init_wait_time = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.628581 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_host_manager = cinder.scheduler.host_manager.HostManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.629011 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_json_config_location = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.629431 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.629856 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_max_attempts = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.630512 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scheduler_weight_handler = cinder.scheduler.weights.OrderedHostWeightHandler {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.631056 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scst_target_driver = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.631496 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] scst_target_iqn_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.631920 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_down_time = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.632351 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] snapshot_name_template = snapshot-%s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.632771 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] snapshot_same_host = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.633190 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] split_loggers = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.633619 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] state_path = /opt/stack/data/cinder {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.634074 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] storage_availability_zone = nova {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.634497 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] storage_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.634931 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] syslog_log_facility = LOG_USER {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.637132 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_helper = lioadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.637774 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.638476 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.639129 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.639783 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] target_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.640417 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] trace_flags = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.641221 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] transfer_api_class = cinder.transfer.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.641909 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] transport_url = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.653573 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] until_refresh = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.654096 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_chap_auth = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.654537 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_default_quota_class = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.654988 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_eventlog = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.655423 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_journal = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.656105 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_json = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.656552 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.657132 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_stderr = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.657702 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] use_syslog = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.658168 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] verify_glance_signatures = enabled {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.658610 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_api_class = cinder.volume.api.API {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.659056 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_backend_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.659482 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_clear = zero {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.659905 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_clear_ionice = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.660329 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_clear_size = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.660752 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.661175 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_copy_bps_limit = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.661615 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_dd_blocksize = 1M {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.662036 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.662586 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_name_template = volume-%s {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.663018 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_number_multiplier = -1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.663448 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volume_usage_audit_period = month {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.663879 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.664316 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] watch_log_file = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Nov 20 20:37:26.664762 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.665206 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.665645 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.enforce_new_defaults = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.666113 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.enforce_scope = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.666650 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.policy_default_rule = default {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.667223 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.667812 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.policy_file = policy.yaml {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.668241 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.668667 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.669110 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.669543 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.669969 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.671814 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_concurrency.disable_process_locking = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.672305 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.672793 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.673249 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.673710 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.674197 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.674645 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.675118 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.connection_string = messaging:// {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.675563 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.676015 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.es_doc_type = notification {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.676460 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.es_scroll_size = 10000 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.676899 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.es_scroll_time = 2m {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.677357 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.filter_error_trace = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.677807 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.hmac_keys = SECRET_KEY {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.678283 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.sentinel_service_name = mymaster {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.678729 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.socket_timeout = 0.1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.679183 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] profiler.trace_sqlalchemy = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.679652 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_password = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.680116 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.680563 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_token = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.681003 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.admin_user = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.682580 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_admin_prefix = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.683189 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.683699 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_port = 35357 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.684153 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_protocol = https {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.684606 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_section = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.685059 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_type = password {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.685510 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_uri = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.685949 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.auth_version = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.686428 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.cache = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.686870 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.687309 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.687799 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.delay_auth_decision = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.688234 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.688696 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.http_connect_timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.689136 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.689583 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.identity_uri = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.690021 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.include_service_catalog = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.690732 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.691876 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.interface = public {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.692332 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.692776 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.693218 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.693657 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.694148 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.694897 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.695568 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.696227 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.696887 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.697663 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.698159 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.region_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.698772 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.699228 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.service_token_roles_required = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.699680 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.service_type = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.700132 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.token_cache_time = 300 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.700708 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.701178 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.backend = dogpile.cache.null {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.702167 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.backend_argument = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.702624 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.config_prefix = cache.oslo {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.703066 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.debug_cache_backend = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.703518 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.703958 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.expiration_time = 600 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.704397 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_dead_retry = 300 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.704923 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.705368 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.705817 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_maxsize = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.706311 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.707117 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.707563 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.memcache_socket_timeout = 1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.708003 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.proxies = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.708444 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_allowed_ciphers = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.708905 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_cafile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.709353 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.709791 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.710268 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cache.tls_keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.710842 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] coordination.backend_url = etcd3+http://10.222.0.41:2379 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.711317 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.backend = sqlalchemy {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.712261 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.712704 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_debug = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.713685 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_parameters = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.714411 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_recycle_time = 3600 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.714856 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.connection_trace = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.715320 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_inc_retry_interval = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.715760 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_max_retries = 20 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.716195 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_max_retry_interval = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.716634 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.db_retry_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.717084 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.max_overflow = 50 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.717534 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.max_pool_size = 5 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.718060 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.max_retries = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.718515 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.mysql_enable_ndb = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.718962 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.719401 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.pool_timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.719858 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.retry_interval = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.720298 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.slave_connection = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.720872 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.sqlite_synchronous = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.721316 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] database.use_db_reconnect = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.722392 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.capabilities = [21] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.722835 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.group = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.723299 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.helper_command = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.723738 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.724184 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.724682 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] privsep_osbrick.user = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.725257 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.725698 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.group = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.726170 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.helper_command = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.726709 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.727146 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.728349 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] cinder_sys_admin.user = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.728349 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.auth_section = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.728349 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.auth_type = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.728977 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.cafile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.729505 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.730023 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.collect_timing = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.730621 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.731306 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.731840 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.send_service_user_token = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.733242 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.split_loggers = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.733723 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] service_user.timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.734341 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.backend_availability_zone = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.734926 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.chap_password = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.735406 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.chap_username = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.735921 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.736488 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_client_cert = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.736969 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_client_cert_key = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.737550 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_data_namespace = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.738095 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.738744 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.739262 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.driver_use_ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.739770 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.enable_unsupported_driver = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.740208 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.740810 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.filter_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.741714 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.goodness_function = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.742870 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.743375 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.743933 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.744948 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.745490 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.746006 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.746464 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_iotype = fileio {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.747044 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.747633 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_target_flags = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.748105 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iscsi_write_cache = on {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.748584 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_helper = tgtadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.749151 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.749715 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.750296 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.750883 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.751582 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.752118 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.num_shell_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.753204 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.753751 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.754311 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.nvmet_port_id = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.754978 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.replication_device = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.755459 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.report_discard_supported = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.755943 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.reserved_percentage = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.756487 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.scst_target_driver = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.757104 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.scst_target_iqn_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.757648 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.storage_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.758240 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_helper = tgtadm {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.758818 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_ip_address = 10.222.0.41 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.759338 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_port = 3260 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.759997 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.760560 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.target_protocol = iscsi {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.761071 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.trace_flags = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.761708 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.unique_fqdn_network = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.762256 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.use_chap_auth = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.763453 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.763922 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_backend_name = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.764391 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_clear = zero {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.764904 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_clear_ionice = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.765383 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_clear_size = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.765828 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.766380 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.766941 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.768329 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.768329 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_reports.file_event_handler = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.768329 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.768602 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_reports.log_dir = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.769094 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.769541 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.770061 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.770577 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.771027 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.771475 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.772061 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.772513 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.773521 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.773973 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.774521 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.774970 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.775420 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.776471 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.777040 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.777587 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.778136 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.778711 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.779188 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.779679 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.780188 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.780995 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.781530 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.782395 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.783524 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.784103 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_rabbit.ssl_version = {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.784642 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.driver = [] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.785162 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.retry = -1 {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.785773 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.786367 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] oslo_messaging_notifications.transport_url = **** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.787105 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.auth_section = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.787605 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.auth_type = password {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.788091 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.788570 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.certfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.789055 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.collect_timing = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.789530 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.insecure = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.790090 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.interface = public {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.790595 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.keyfile = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.791472 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.region_name = RegionOne {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.792069 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.split_loggers = False {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.792756 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.timeout = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.793853 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] nova.token_auth_url = None {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Nov 20 20:37:26.794399 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.service [None req-3c49bc9b-58e5-4257-9921-08bca1154e26 None None] ******************************************************************************** {{(pid=133499) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Nov 20 20:37:26.795227 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.service [-] Starting cinder-scheduler node (version 19.1.0) Nov 20 20:37:26.804238 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.manager [None req-cc545fc7-8b41-4678-b261-c877a549172f None None] Initiating service 1 cleanup Nov 20 20:37:26.810785 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.manager [None req-cc545fc7-8b41-4678-b261-c877a549172f None None] Service 1 cleanup completed. Nov 20 20:37:26.811409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-cc545fc7-8b41-4678-b261-c877a549172f None None] Creating RPC server for service cinder-scheduler {{(pid=133499) start /opt/stack/cinder/cinder/service.py:231}} Nov 20 20:37:26.818287 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.service [None req-cc545fc7-8b41-4678-b261-c877a549172f None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=133499) start /opt/stack/cinder/cinder/service.py:237}} Nov 20 20:37:28.478502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-eb301526-e96f-4ad8-be1b-71209b0f26ce None None] Received backup service update from ncdv-master-1: {'backend_state': False, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:37:30.361774 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-16d76181-154b-41b3-8c01-56389bb1e69e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:37:30.372836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-16d76181-154b-41b3-8c01-56389bb1e69e None None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 37, 30, 324212)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:38:08.964881 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.periodic_task [None req-bae568b5-b0d1-4420-846e-95b21dc01492 None None] Running periodic task SchedulerManager._clean_expired_messages {{(pid=133499) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Nov 20 20:38:08.969984 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.message.api [None req-bae568b5-b0d1-4420-846e-95b21dc01492 None None] Deleted 0 expired messages. Nov 20 20:38:08.970351 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_service.periodic_task [None req-bae568b5-b0d1-4420-846e-95b21dc01492 None None] Running periodic task SchedulerManager._clean_expired_reservation {{(pid=133499) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Nov 20 20:38:54.556354 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-272aa34f-6922-4efb-910f-68546088d85c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:39:02.586929 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-55bd002b-0778-4ca3-a442-2197c39c2987 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:39:54.552723 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-74e15787-0730-4788-9c62-853f8609cabd None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:40:02.591210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8ad75dd0-7a94-4a29-aa39-ef590b35c406 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:40:54.551107 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-371eca4a-23ee-43b9-ad24-347da5c19830 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:41:02.589457 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-076a094d-e669-4f28-8388-852bf5633bbb None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:41:54.555787 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-881dc056-18cf-4360-bb04-dd6b897590f1 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:42:02.588845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-873e5a25-6c0d-4038-b706-ed90f40fc072 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:42:54.560711 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6e281806-d76e-4821-a8f2-fc34fba3901f None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:43:02.586515 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4f87fa58-0036-417b-ae5b-7d465c1a1e14 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:43:54.565558 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0caf9d2a-dd3f-410a-99a1-165d261a2b3d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:44:02.588939 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0cb67a48-9458-4a02-9658-174ee1a98e97 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:44:54.570468 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8a183182-a4cc-45a7-bee9-59a1be6d70e1 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:45:02.591391 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-957969d0-6a31-4d45-a11c-8af3721a0fe7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:45:54.577254 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8868ab03-bee3-4fc5-98c8-e22f1f96f925 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:46:02.723534 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a7ca6b3-ff5d-436a-a98e-9a267cd20baf None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:46:54.574603 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4c6a2ace-1a00-49d4-a07e-c28c5561d42f None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:47:02.710373 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9c6989cb-87bb-47b4-9031-879f08f67509 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:47:03.414408 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Flow 'volume_create_scheduler' (8c4a9b80-f499-49de-a3f3-7f708c8fdcf4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:47:03.419098 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (38a32220-1367-41b9-854d-9eeece808d97) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:47:03.422537 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (38a32220-1367-41b9-854d-9eeece808d97) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(78f257d6-df0c-48f8-9161-47c8bf8f19dd),volume_id=78f257d6-df0c-48f8-9161-47c8bf8f19dd,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:47:03.425046 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (87223b5a-ad0b-488a-b14e-bc23c9fddaa4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:47:03.435807 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:47:03.471530 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 47, 2, 701773)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:47:03.472233 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:47:03.472947 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:47:03.473610 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (78f257d6-df0c-48f8-9161-47c8bf8f19dd) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:47:03.474204 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.93, total_capacity_gb: 19.93,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:47:02.701773 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:47:03.578760 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:47:03.579066 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:47:03.579509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.93, total_capacity_gb: 19.93,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:47:02.701773] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:47:03.580759 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Weigher CapacityWeigher returned, weigher value is {max: 398.6, min: 398.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:47:03.581161 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:47:03.581999 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:47:03.582425 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:47:03.582028 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:47:03.770726 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (87223b5a-ad0b-488a-b14e-bc23c9fddaa4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:47:03.774578 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b3d21236-4945-4976-98e2-6dcc68ad7415 tempest-VolumesAdminNegativeTest-1918384537 None] Flow 'volume_create_scheduler' (8c4a9b80-f499-49de-a3f3-7f708c8fdcf4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:47:17.639477 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Flow 'volume_create_scheduler' (2f50c3fb-21a2-4f56-a5ce-e73e645d9e72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:47:17.654631 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e9af31c4-8c0d-4fa8-acd5-dfeb85b3ef4d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:47:17.657233 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e9af31c4-8c0d-4fa8-acd5-dfeb85b3ef4d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(6ddf0992-3ec0-4aad-b514-6aba5d937841),volume_id=6ddf0992-3ec0-4aad-b514-6aba5d937841,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:47:17.674127 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e871df1b-a4ba-44c0-b456-0ecec4fb5d1f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:47:17.700540 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:47:17.737710 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 47, 2, 701773)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:47:17.737710 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:47:17.739076 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:47:17.739642 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (6ddf0992-3ec0-4aad-b514-6aba5d937841) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:47:17.740010 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:47:03.582028 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: from cryptography.utils import int_from_bytes Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: from cryptography.utils import int_from_bytes Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: /opt/stack/cinder/cinder/db/sqlalchemy/models.py:138: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: last_heartbeat = column_property( Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: /opt/stack/cinder/cinder/db/sqlalchemy/models.py:144: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: num_hosts = column_property( Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: /opt/stack/cinder/cinder/db/sqlalchemy/models.py:151: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: num_down_hosts = column_property( Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:47:17.741900 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:47:17.743496 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:47:17.745258 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 398.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:47:17.747164 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:47:17.748909 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:47:17.748909 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:47:17.748909 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:47:17.748909 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:47:17.748909 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:47:03.582028] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:47:17.748909 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Weigher CapacityWeigher returned, weigher value is {max: 397.6, min: 397.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:47:17.749261 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:47:17.749261 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:47:17.749261 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:47:17.746018 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:47:17.820620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e871df1b-a4ba-44c0-b456-0ecec4fb5d1f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:47:17.826298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4c83080f-b4e6-436a-bacf-985bbfbb04d9 tempest-VolumesAdminNegativeTest-1918384537 None] Flow 'volume_create_scheduler' (2f50c3fb-21a2-4f56-a5ce-e73e645d9e72) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:47:20.278695 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-98e6a5dc-a911-44a3-ba8d-2a576609b4cb tempest-VolumesAdminNegativeTest-1918384537 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:47:21.518856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-75556e1e-d38a-4d1b-a9d9-231c6650a5aa tempest-VolumesAdminNegativeTest-1918384537 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:47:54.575323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3dcfb7c4-712b-49b6-8b26-1cc646eb6419 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:48:02.725153 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e7280e97-e410-411d-a21b-7cb1d91fa7f0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:48:54.580657 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c62838cf-c5f7-48df-98a0-d1ec34d219a3 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:49:02.658496 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-68eb536d-ce0b-4f6b-9e3f-b99a71c6b115 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:49:11.231856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Flow 'volume_create_scheduler' (de4d9984-8092-4d01-969e-c6671b80af65) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:11.235165 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (41890634-21a5-4a84-ab73-2b3f52efe5da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:11.237714 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (41890634-21a5-4a84-ab73-2b3f52efe5da) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f3a25071-9204-456d-ad65-141e2d6b0743),volume_id=f3a25071-9204-456d-ad65-141e2d6b0743,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:11.240298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3077062c-32ba-44c6-8e2a-890bdfc1addb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:11.248471 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:49:11.262948 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 49, 2, 649544)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:49:11.263731 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:49:11.263821 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:11.264190 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f3a25071-9204-456d-ad65-141e2d6b0743) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:49:11.264516 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.93, total_capacity_gb: 19.93,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:02.649544 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:49:11.265589 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:49:11.267205 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 378.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:49:11.268576 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:49:11.271446 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:49:11.273365 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:11.273365 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:11.273365 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.93, total_capacity_gb: 19.93,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:02.649544] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:49:11.273365 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Weigher CapacityWeigher returned, weigher value is {max: 398.6, min: 398.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:49:11.273679 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:49:11.273679 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:49:11.273679 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:11.267936 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:49:11.304547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3077062c-32ba-44c6-8e2a-890bdfc1addb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:11.307763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2873d4f7-63af-4786-bdd2-71377f73d280 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Flow 'volume_create_scheduler' (de4d9984-8092-4d01-969e-c6671b80af65) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:32.017825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Flow 'volume_create_scheduler' (c1020098-0ca5-4bdb-a7e8-aefc46937a8f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:32.023615 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (bd333d9f-9578-45d9-a367-102acd5bdedc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:32.027080 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (bd333d9f-9578-45d9-a367-102acd5bdedc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b7841b2e-06ba-4d66-a2fd-bd25a876b43c),volume_id=b7841b2e-06ba-4d66-a2fd-bd25a876b43c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:32.029629 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9032d0c5-b4d0-47de-b832-949b26145bbe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:32.038532 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:49:32.087029 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 49, 2, 649544)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:49:32.087437 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:49:32.087812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:32.088183 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b7841b2e-06ba-4d66-a2fd-bd25a876b43c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:49:32.088862 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:11.267936 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:49:32.089949 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 398.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:49:32.091436 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:49:32.093512 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:49:32.098289 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:49:32.101471 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:49:32.101471 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:32.101471 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:32.101471 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:11.267936] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:49:32.101471 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Weigher CapacityWeigher returned, weigher value is {max: 397.6, min: 397.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:49:32.102201 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:49:32.102201 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:49:32.102201 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:32.092540 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:49:32.131275 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9032d0c5-b4d0-47de-b832-949b26145bbe) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:32.136140 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-010050a3-4fb2-4826-954e-d6c6f6660a7b tempest-ServersTestBootFromVolume-1177398021 None] Flow 'volume_create_scheduler' (c1020098-0ca5-4bdb-a7e8-aefc46937a8f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:33.225325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Flow 'volume_create_scheduler' (8656b424-c4a3-4eab-bddd-db4f5e6d5ad0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:33.227768 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (51a88ed2-1b4a-4f8f-83d6-de9759dc0019) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:33.230261 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (51a88ed2-1b4a-4f8f-83d6-de9759dc0019) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(609b9a90-c344-44ec-95a5-ae6618ae18dd),volume_id=609b9a90-c344-44ec-95a5-ae6618ae18dd,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:33.232607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (db3b87ce-1d0e-4e18-8ae7-cc1214804576) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:33.246609 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:49:33.270969 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 49, 2, 649544)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:49:33.271721 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:49:33.272527 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:33.273828 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (609b9a90-c344-44ec-95a5-ae6618ae18dd) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:49:33.274838 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:32.092540 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:49:33.276633 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:33.277404 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:33.277842 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:32.092540] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:49:33.278432 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Weigher CapacityWeigher returned, weigher value is {max: 396.6, min: 396.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:49:33.278791 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:49:33.279269 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:49:33.279649 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.93, total_capacity_gb: 19.93,allocated_capacity_gb: 3, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 3,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:33.279296 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:49:33.315881 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (db3b87ce-1d0e-4e18-8ae7-cc1214804576) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:33.320178 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1fdb609a-4a6b-4334-b340-9413f53f5e05 tempest-LiveMigrationTest-514169898 None] Flow 'volume_create_scheduler' (8656b424-c4a3-4eab-bddd-db4f5e6d5ad0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:48.626298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Flow 'volume_create_scheduler' (dacd70b7-d46f-451b-b710-1be5c1c97b49) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:48.637846 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (744850ab-ec43-4349-ae65-8b709f695a11) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:48.640714 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (744850ab-ec43-4349-ae65-8b709f695a11) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0803734f-6742-4a4d-8758-d341c66337d2),volume_id=0803734f-6742-4a4d-8758-d341c66337d2,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:48.642837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2c54949-d51e-4a60-a85e-5a10025daf6d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:48.669885 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:49:48.687357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 49, 2, 649544)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:49:48.687908 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:49:48.688475 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:48.689117 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0803734f-6742-4a4d-8758-d341c66337d2) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:49:48.690417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.93, total_capacity_gb: 19.93,allocated_capacity_gb: 3, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 3,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:33.279296 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 378.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:49:48.692286 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:49:48.694583 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:49:48.696612 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 358.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:49:48.698360 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:49:48.700553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:48.700553 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:48.701914 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:48.702580 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:48.703142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.93, total_capacity_gb: 19.93,allocated_capacity_gb: 3, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 3,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:33.279296] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:49:48.704378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Weigher CapacityWeigher returned, weigher value is {max: 395.6, min: 395.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:49:48.704920 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:49:48.705642 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:49:48.706313 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.93, total_capacity_gb: 19.93,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:48.705857 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:49:48.747026 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2c54949-d51e-4a60-a85e-5a10025daf6d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:48.754785 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-70240858-1d0f-4453-9715-610ab472dbd2 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Flow 'volume_create_scheduler' (dacd70b7-d46f-451b-b710-1be5c1c97b49) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:53.593853 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Flow 'volume_create_scheduler' (c768f993-6f9d-4487-8a0e-5900140ca3f1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:53.607731 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (175de255-454b-4253-a74f-a63793565fbb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:53.620644 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (175de255-454b-4253-a74f-a63793565fbb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(6e2b1f6d-f606-4f0c-a332-219fb5652d9f),volume_id=6e2b1f6d-f606-4f0c-a332-219fb5652d9f,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:53.632720 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c26a864d-d7c3-4dba-b4f8-a105d3f46cf5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:49:53.659244 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:49:53.700401 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 49, 2, 649544)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:49:53.700967 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:49:53.701427 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:53.701994 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (6e2b1f6d-f606-4f0c-a332-219fb5652d9f) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:49:53.702676 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.93, total_capacity_gb: 19.93,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:48.705857 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:49:53.704315 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:49:53.707826 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 338.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:49:53.709242 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:49:53.711092 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:49:53.714318 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:53.714958 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:49:53.715541 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.93, total_capacity_gb: 19.93,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:48.705857] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:49:53.716232 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Weigher CapacityWeigher returned, weigher value is {max: 394.6, min: 394.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:49:53.716797 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:49:53.717567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:49:53.718361 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.93, total_capacity_gb: 19.93,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:49:53.717877 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:49:53.749879 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c26a864d-d7c3-4dba-b4f8-a105d3f46cf5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:49:53.752906 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dc78d417-7286-446d-b4aa-fcad7e59c2be tempest-LiveMigrationTest-514169898 None] Flow 'volume_create_scheduler' (c768f993-6f9d-4487-8a0e-5900140ca3f1) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:49:54.583915 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3fa8bbe7-6c52-4067-957f-f53ea7e37a93 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:50:02.784138 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-67435741-594f-461b-a5e5-046c6920ee6e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:50:05.068353 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Flow 'volume_create_scheduler' (e5cad3f8-d747-4d90-9f4e-f82e541ed72b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:50:05.085100 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3b07fd3b-4a15-42ce-8fe7-aca6b780fd8b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:50:05.089271 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3b07fd3b-4a15-42ce-8fe7-aca6b780fd8b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(506c87e8-8c5a-4446-b990-9916d5078cd5),volume_id=506c87e8-8c5a-4446-b990-9916d5078cd5,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:50:05.091444 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e9eb3f8c-c7e5-47e9-931a-728d4b707871) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:50:05.124147 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:50:05.171885 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 50, 2, 777633)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:50:05.172320 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:50:05.172883 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:05.173102 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (506c87e8-8c5a-4446-b990-9916d5078cd5) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:50:05.173560 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.93, total_capacity_gb: 19.93,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:02.777633 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:50:05.174896 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 318.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:50:05.179942 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:50:05.181808 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:50:05.187053 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:50:05.194409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:05.194409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:05.194409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.93, total_capacity_gb: 19.93,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:02.777633] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:50:05.194409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Weigher CapacityWeigher returned, weigher value is {max: 393.6, min: 393.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:50:05.194409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:50:05.195256 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:50:05.195256 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:05.181467 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:50:05.238403 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e9eb3f8c-c7e5-47e9-931a-728d4b707871) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:50:05.243978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-540e8309-d9f0-4291-abcf-34145d5ea2aa tempest-TaggedBootDevicesTest_v242-1749553438 None] Flow 'volume_create_scheduler' (e5cad3f8-d747-4d90-9f4e-f82e541ed72b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:50:14.477290 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Flow 'volume_create_scheduler' (0cb301d8-c7eb-432b-ba47-f4df12be473a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:50:14.480551 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (5d33d2a3-b5c8-4d02-b4b2-e2510ebbdc15) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:50:14.483080 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (5d33d2a3-b5c8-4d02-b4b2-e2510ebbdc15) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c3e7b33d-1239-466a-aa16-3ff43c1eea01),volume_id=c3e7b33d-1239-466a-aa16-3ff43c1eea01,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:50:14.485079 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (583ecba5-234b-44fb-9505-afb24fec623c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:50:14.493002 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:50:14.522024 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 50, 2, 777633)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:50:14.522445 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:50:14.522813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:14.523172 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c3e7b33d-1239-466a-aa16-3ff43c1eea01) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:50:14.523568 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:05.181467 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:50:14.534979 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:14.537542 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:14.537856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.93, total_capacity_gb: 19.93,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:05.181467] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:50:14.538318 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Weigher CapacityWeigher returned, weigher value is {max: 392.6, min: 392.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:50:14.538675 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:50:14.539156 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:50:14.539544 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:14.539185 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:50:14.609172 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (583ecba5-234b-44fb-9505-afb24fec623c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:50:14.611952 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0d87e636-2524-456f-b4d8-73bfdd72311b tempest-TaggedBootDevicesTest_v242-1749553438 None] Flow 'volume_create_scheduler' (0cb301d8-c7eb-432b-ba47-f4df12be473a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:50:16.103383 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Flow 'volume_create_scheduler' (a7ffb5c8-88b1-4f5c-99e8-88554a5061fd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:50:16.111152 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e2684b87-eaab-425b-8c27-ed4176661354) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:50:16.118914 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e2684b87-eaab-425b-8c27-ed4176661354) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(71143215-11e7-4aed-8258-31185f04987e),volume_id=71143215-11e7-4aed-8258-31185f04987e,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:50:16.121005 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3f9ebe2d-2934-4491-83c4-75531a6066c7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:50:16.150240 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:50:16.182701 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.93, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 50, 2, 777633)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:50:16.182965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:50:16.183334 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:16.183686 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (71143215-11e7-4aed-8258-31185f04987e) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:50:16.184089 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:14.539185 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 398.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:50:16.185152 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:50:16.187348 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:50:16.189925 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 378.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:50:16.191591 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:50:16.194341 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:50:16.194341 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:50:16.194341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:16.194341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:50:16.194341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.93, total_capacity_gb: 19.93,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:14.539185] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:50:16.194341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Weigher CapacityWeigher returned, weigher value is {max: 391.6, min: 391.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:50:16.194843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:50:16.194843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:50:16.197189 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.93, total_capacity_gb: 19.93,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:50:16.194665 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:50:16.256600 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3f9ebe2d-2934-4491-83c4-75531a6066c7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:50:16.258892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e17e6051-16b5-428c-b31a-b3f2d6c4473a tempest-TaggedBootDevicesTest_v242-1749553438 None] Flow 'volume_create_scheduler' (a7ffb5c8-88b1-4f5c-99e8-88554a5061fd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:50:51.405416 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-9d5e0f74-1ef3-4fcd-b5ed-e77c5b2899c5 req-602343cc-b6ac-4654-819d-d0fdb7e3b4b7 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.47, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:50:52.249380 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ef47a84d-1505-494e-82c4-414ca6408f80 tempest-LiveAutoBlockMigrationV225Test-1439503069 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.47, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:50:54.582737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ccdc5619-9097-41e6-97f1-8a58b99f2221 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:51:02.705181 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0d92c1f8-f630-44ed-a96f-72979ec189a1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.47, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:51:07.660486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-89496f55-ffaa-4c5d-8356-7ddaaaa3c701 req-27547710-5ee8-46c6-9a5e-e7f139093675 tempest-LiveMigrationTest-514169898 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.46, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:51:07.965985 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f3e9e71c-5f65-4398-8b12-0740c5ef7503 tempest-LiveMigrationTest-514169898 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.46, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:51:54.580052 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d10160be-b33f-451a-aec7-6a9dd3e053b8 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:52:02.638615 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3cb3e1e3-a60a-4d3f-961e-8266b5c6fabf None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.89, 'free_capacity_gb': 19.4, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:52:54.578829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-804b4a55-fd39-4005-8d15-ed744fe0c453 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:52:59.823770 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-6d45cd97-47b9-47ad-abe3-c62f86c0447e req-bc835e60-7426-4754-85ba-5353b255d54f tempest-ServersTestBootFromVolume-1177398021 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.86, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:53:02.686523 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d10afe0f-f696-4037-8dfc-171a61030014 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.86, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:53:54.578098 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-37ae494e-3da1-4b5a-b088-ddbd9d3a1deb None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:54:02.672306 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cc6faae7-caa3-4eb4-99eb-2438c2b5cef3 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.29, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:54:54.575959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a425eb1d-ef1b-4b94-8802-83c4ffcd2a56 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:55:02.672093 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-83bfcfdd-beb4-4b07-91b6-088f43da5119 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:55:46.818807 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Flow 'volume_create_scheduler' (bf03f61b-b4f0-4c69-88b4-185d3f0e131a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:55:46.823199 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (73aff1c5-a3fe-43e1-883b-314712ae1fe7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:55:46.834168 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (73aff1c5-a3fe-43e1-883b-314712ae1fe7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5fbc117a-9caa-41b6-b943-e32e4d68e170),volume_id=5fbc117a-9caa-41b6-b943-e32e4d68e170,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:55:46.840224 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c71a5c0a-b667-4286-bf37-fb55408c1905) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:55:46.848745 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:55:46.869274 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 55, 2, 658972)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:55:46.869914 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:55:46.870757 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:55:46.871268 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5fbc117a-9caa-41b6-b943-e32e4d68e170) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:55:46.872022 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:55:02.658972 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:55:46.873391 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:55:46.874683 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 358.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:55:46.876021 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:55:46.877428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:55:46.880604 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:55:46.881347 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:55:46.881888 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:55:46.882538 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:55:02.658972] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:55:46.883120 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Weigher CapacityWeigher returned, weigher value is {max: 392.8, min: 392.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:55:46.883693 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:55:46.884384 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:55:46.884900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:55:46.884543 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:55:46.923494 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c71a5c0a-b667-4286-bf37-fb55408c1905) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:55:46.926350 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bfbb8d5e-afd0-49e3-aff7-2a8ddc838cca tempest-DeleteServersTestJSON-1053652199 None] Flow 'volume_create_scheduler' (bf03f61b-b4f0-4c69-88b4-185d3f0e131a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:55:54.575691 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7548dd90-6f52-46fc-959e-00aebbe90224 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:56:02.740296 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ae69b691-7c51-49b4-b728-6e7488804aad None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:56:14.165532 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Flow 'volume_create_scheduler' (593ee3d7-ff54-4d02-aa53-083e9e67bb1a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:56:14.176233 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1d30c4f1-3736-4dbd-811e-d4ed759fec9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:56:14.184620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1d30c4f1-3736-4dbd-811e-d4ed759fec9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ef6864fa-67a2-4014-a8ed-07b678b6ff48),volume_id=ef6864fa-67a2-4014-a8ed-07b678b6ff48,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:56:14.192113 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (63aa53bd-8780-41b2-93f5-86f05e3032a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:56:14.205830 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:56:14.247331 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 56, 2, 734894)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:56:14.248275 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:56:14.248902 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:56:14.249457 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ef6864fa-67a2-4014-a8ed-07b678b6ff48) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:56:14.250275 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:56:02.734894 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:56:14.281393 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:56:14.283067 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:56:14.284858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:56:14.286500 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:56:14.288607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:56:14.288607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:56:14.288607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:56:02.734894] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:56:14.288607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:56:14.288607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:56:14.289085 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:56:14.289085 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:56:14.287697 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:56:14.385648 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (63aa53bd-8780-41b2-93f5-86f05e3032a5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:56:14.397181 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-298411ea-b49c-4241-aad9-5d56184444b5 tempest-ServerRescueNegativeTestJSON-204801631 None] Flow 'volume_create_scheduler' (593ee3d7-ff54-4d02-aa53-083e9e67bb1a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:56:48.045655 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Flow 'volume_create_scheduler' (f933ccbe-e725-42f6-8729-84be59f73583) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:56:48.049942 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3d7e5de9-2874-433a-a631-c18f18d3c4bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:56:48.058464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3d7e5de9-2874-433a-a631-c18f18d3c4bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(402ed45c-bf42-47f0-9c9a-8a03020d88ea),volume_id=402ed45c-bf42-47f0-9c9a-8a03020d88ea,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:56:48.061482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cabb7a3a-4211-40d1-bf29-39a50360bd01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:56:48.070729 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:56:48.108486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 56, 2, 734894)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:56:48.109217 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:56:48.109869 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:56:48.111041 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (402ed45c-bf42-47f0-9c9a-8a03020d88ea) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:56:48.111754 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:56:14.287697 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:56:48.113737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:56:48.114404 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:56:48.115265 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:56:14.287697] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:56:48.116360 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Weigher CapacityWeigher returned, weigher value is {max: 390.8, min: 390.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:56:48.116904 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:56:48.117590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:56:48.118209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:56:48.117805 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:56:48.176459 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cabb7a3a-4211-40d1-bf29-39a50360bd01) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:56:48.181914 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dbd736c1-6fc2-4de8-b06c-6cb2ef0f01e4 tempest-ServerRescueNegativeTestJSON-204801631 None] Flow 'volume_create_scheduler' (f933ccbe-e725-42f6-8729-84be59f73583) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:56:54.580734 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6da8ed08-ec48-4387-800c-bbd3e29f2a69 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:57:02.676359 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1ca9c900-2141-4d1c-a17f-9c4424721838 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:57:07.881987 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Flow 'volume_create_scheduler' (d3c0f03e-5c6c-4e1f-9fd3-b039ba027c43) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:07.888383 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3d2e3d44-5d96-4abd-8fd8-192e4858be0c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:07.892012 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3d2e3d44-5d96-4abd-8fd8-192e4858be0c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0687cab0-7083-48c0-afcd-34197a5e0e13),volume_id=0687cab0-7083-48c0-afcd-34197a5e0e13,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:07.894926 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (86392cba-71c2-4a09-98e0-c78491a1b71f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:07.906845 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:57:07.932361 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 57, 2, 670779)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:57:07.932778 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:57:07.933183 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:07.933806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0687cab0-7083-48c0-afcd-34197a5e0e13) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:57:07.934551 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:02.670779 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:57:07.935843 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:57:07.937705 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:57:07.939483 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:57:07.941257 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:57:07.945178 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:07.945723 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:07.946270 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:02.670779] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:57:07.946978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Weigher CapacityWeigher returned, weigher value is {max: 389.8, min: 389.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:57:07.947547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:57:07.948168 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:57:07.948684 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:07.948333 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:57:08.001317 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (86392cba-71c2-4a09-98e0-c78491a1b71f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:08.003078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5a060226-db5e-4be4-8455-3b09d7214c8a tempest-AttachSCSIVolumeTestJSON-782544545 None] Flow 'volume_create_scheduler' (d3c0f03e-5c6c-4e1f-9fd3-b039ba027c43) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:22.947066 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-76d2bcca-542d-4b9f-90e0-cfcc958522db tempest-AttachSCSIVolumeTestJSON-782544545 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:57:26.252867 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6bbd239c-a96b-46d6-bf34-1cc9ca5dcad1 tempest-ServerRescueNegativeTestJSON-204801631 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:57:27.796803 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d08e54a7-9f6c-4a23-adac-a78cc269002b tempest-ServerRescueNegativeTestJSON-204801631 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:57:54.575148 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f48cf55f-ee79-434b-8d55-49157edf45f3 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:57:55.420819 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Flow 'volume_create_scheduler' (942f1379-dd62-4556-9295-411bf30fb5da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:55.430754 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (535e0dc8-6212-43c1-99b9-34d2e1261e8d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:55.435176 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (535e0dc8-6212-43c1-99b9-34d2e1261e8d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(eab55441-b110-401d-967c-ff260090887c),volume_id=eab55441-b110-401d-967c-ff260090887c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:55.440693 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f7a317d0-167e-4425-8d72-c7a3902a9e20) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:55.460080 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:57:55.482099 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 57, 27, 791452)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:57:55.482785 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:57:55.483354 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:55.484624 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (eab55441-b110-401d-967c-ff260090887c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:57:55.485538 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:27.791452 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:57:55.487027 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:57:55.489072 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:57:55.490506 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:57:55.492265 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:57:55.493570 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:57:55.498308 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:55.501915 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:55.502499 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:27.791452] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:57:55.503521 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:57:55.504580 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:57:55.508396 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:57:55.509056 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:55.508681 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:57:55.574203 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f7a317d0-167e-4425-8d72-c7a3902a9e20) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:55.580300 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1d4c5811-4e34-4b06-a1a6-2f8aabaaeefe tempest-VolumesTestJSON-430761339 None] Flow 'volume_create_scheduler' (942f1379-dd62-4556-9295-411bf30fb5da) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:57.485044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Flow 'volume_create_scheduler' (04e3f9cd-af46-47e7-9f6e-e6d0f559ded6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:57.495380 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (64ff9be8-cded-44bd-b398-d8a13c8f4e4a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:57.498281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (64ff9be8-cded-44bd-b398-d8a13c8f4e4a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(db054416-9aec-4e4b-85aa-14e648e12ef5),volume_id=db054416-9aec-4e4b-85aa-14e648e12ef5,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:57.499913 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8023e597-176d-4cf5-960f-5e93d8bf27de) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:57.524043 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:57:57.543174 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 57, 27, 791452)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:57:57.543745 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:57:57.544283 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:57.544793 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (db054416-9aec-4e4b-85aa-14e648e12ef5) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:57:57.545342 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:55.508681 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:57:57.546657 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:57:57.548198 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:57:57.549980 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:57:57.551975 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:57:57.554460 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:57.555044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:57.555866 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:55.508681] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:57:57.556429 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Weigher CapacityWeigher returned, weigher value is {max: 390.8, min: 390.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:57:57.556927 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:57:57.558138 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:57:57.558720 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:57.558363 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:57:57.627736 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8023e597-176d-4cf5-960f-5e93d8bf27de) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:57.637819 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-37698ae4-d849-4e4c-b835-17057dd84427 tempest-VolumesTestJSON-430761339 None] Flow 'volume_create_scheduler' (04e3f9cd-af46-47e7-9f6e-e6d0f559ded6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:59.249080 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Flow 'volume_create_scheduler' (789b67fc-520b-40fe-a57e-31edc060f3d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:57:59.252573 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fb2210db-f527-4551-bc95-876db02baf03) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:59.258331 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fb2210db-f527-4551-bc95-876db02baf03) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9d45cf2c-0b35-460d-b681-0db7f4ba2518),volume_id=9d45cf2c-0b35-460d-b681-0db7f4ba2518,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:59.262619 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ae8025ad-3e21-44c3-9d5e-111fdff36c4c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:57:59.271864 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:57:59.289122 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 57, 27, 791452)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:57:59.289556 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:57:59.290017 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:59.290480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9d45cf2c-0b35-460d-b681-0db7f4ba2518) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:57:59.290930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:57.558363 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:57:59.292500 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:59.293089 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:57:59.293746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:57.558363] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:57:59.294281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Weigher CapacityWeigher returned, weigher value is {max: 389.8, min: 389.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:57:59.294805 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:57:59.296098 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:57:59.296635 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.84,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:57:59.296278 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:57:59.342164 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ae8025ad-3e21-44c3-9d5e-111fdff36c4c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:57:59.344263 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7d8e64b7-e4c7-4f3e-b110-cd661877911f tempest-VolumesTestJSON-430761339 None] Flow 'volume_create_scheduler' (789b67fc-520b-40fe-a57e-31edc060f3d2) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:58:02.686148 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3581b08a-8549-4e44-804d-3ddd805b51a7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:58:04.666661 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bf6d617a-926f-4b66-a9ca-e8334972fb1a tempest-VolumesTestJSON-430761339 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:58:05.872367 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-723d08c9-dcb0-4a09-bc93-77d31883adb6 tempest-VolumesTestJSON-430761339 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:58:07.298597 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c21e3c59-d5f0-4ac0-9812-e4decf04f663 tempest-VolumesTestJSON-430761339 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:58:54.574343 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8c72ceb2-4884-4ed6-9b0e-9453ec776a57 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:59:02.633922 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4db8d4dc-99b7-4e1f-a64b-8e8594fdc317 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:59:31.041422 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f6bc09df-a3d8-4fba-aec8-0f4fa40bc724 tempest-VolumePoolsAdminTestsJSON-574539802 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 59, 2, 629726)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:59:31.103375 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f5fa194e-fe93-452d-aba0-1d6af94a3bff tempest-VolumePoolsAdminTestsJSON-574539802 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 59, 2, 629726)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:59:32.658660 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9176915b-a598-4125-83ce-520bcc7998df tempest-DeleteServersTestJSON-1053652199 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 20:59:50.988689 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Flow 'volume_create_scheduler' (1190151a-0b4c-40c0-9e8c-9007c2e431fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:59:50.991263 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9538a02d-ef6d-4e0e-9885-1e3a0136762a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:59:50.993744 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9538a02d-ef6d-4e0e-9885-1e3a0136762a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b58761a2-25fc-458b-b255-11afa9b13971),volume_id=b58761a2-25fc-458b-b255-11afa9b13971,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:59:50.995745 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9f477e0e-62aa-4df6-bae9-6e9c71265328) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 20:59:51.015127 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 20:59:51.070406 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 59, 32, 637765)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:59:51.070835 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:59:51.071202 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:59:51.071564 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b58761a2-25fc-458b-b255-11afa9b13971) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:59:51.071991 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:59:32.637765 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:59:51.073123 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 20:59:51.075452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 20:59:51.078382 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 20:59:51.081957 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 20:59:51.085898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:59:51.085898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:59:51.085898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:59:32.637765] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:59:51.085898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Weigher CapacityWeigher returned, weigher value is {max: 392.8, min: 392.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:59:51.085898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:59:51.086788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 20:59:51.086788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:59:51.075511 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:59:51.151033 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9f477e0e-62aa-4df6-bae9-6e9c71265328) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 20:59:51.153184 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe40a191-a38d-4832-9678-2e40a8ac7df1 tempest-SnapshotMetadataTestJSON-81589040 None] Flow 'volume_create_scheduler' (1190151a-0b4c-40c0-9e8c-9007c2e431fa) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 20:59:52.618134 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 20, 59, 32, 637765)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 20:59:52.619981 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 20:59:52.620350 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:59:52.622806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 20:59:52.624581 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:59:51.075511 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 20:59:52.633892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:59:52.635961 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 20:59:52.637161 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:59:51.075511] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 20:59:52.638464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 20:59:52.640849 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 20:59:52.642464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-36c3cc5e-91a8-4bfd-8066-ae9beb04d4a9 tempest-SnapshotMetadataTestJSON-81589040 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 20:59:52.642090 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 20:59:54.576201 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-87cc749f-be02-498f-a876-dfd813b0bbf5 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 20:59:57.284207 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-74df4a65-3232-42d7-9463-c9adc4617d78 tempest-SnapshotMetadataTestJSON-81589040 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:00:02.757615 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c7e4b2f2-e070-4a89-8ffa-571efbcfbc5d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:00:04.767732 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Flow 'volume_create_scheduler' (2ec8dae2-af6f-4eb4-8a90-d67d8acd928f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:04.773841 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (41236e31-f77d-4c8b-a6ab-c8a11f0c0aba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:04.782402 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (41236e31-f77d-4c8b-a6ab-c8a11f0c0aba) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(3190d69e-dd1f-4670-ab4d-f86068db2f63),volume_id=3190d69e-dd1f-4670-ab4d-f86068db2f63,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:04.789521 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b4e13079-cab7-44ee-832e-f8082c3ca52b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:04.807778 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:00:04.846793 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 0, 2, 752718)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:00:04.848978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:00:04.849546 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:04.850893 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (3190d69e-dd1f-4670-ab4d-f86068db2f63) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:00:04.851503 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:02.752718 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:00:04.856449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:04.858078 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:00:04.859604 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:00:04.861807 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:00:04.867417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:04.867809 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:04.868248 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:02.752718] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:00:04.868874 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Weigher CapacityWeigher returned, weigher value is {max: 392.8, min: 392.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:00:04.872501 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:00:04.872917 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:00:04.873314 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:04.872946 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:00:04.927622 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b4e13079-cab7-44ee-832e-f8082c3ca52b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:04.941336 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e68d3602-9df8-4fed-9d1b-1a6eea86b15a tempest-VolumesDeleteCascade-379077449 None] Flow 'volume_create_scheduler' (2ec8dae2-af6f-4eb4-8a90-d67d8acd928f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:06.483441 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 0, 2, 752718)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:00:06.487294 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:00:06.488802 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:06.489356 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:00:06.490856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:04.872946 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:00:06.500384 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:00:06.502335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:06.503893 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:00:06.505622 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:06.507430 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:06.507430 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:06.508281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:06.510873 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:06.513337 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:04.872946] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:00:06.514558 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:00:06.517096 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:00:06.522629 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e55b7bfa-e7f6-4d75-8cf2-51fef70a4844 tempest-VolumesDeleteCascade-379077449 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:06.520063 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:00:08.921229 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 0, 2, 752718)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:00:08.922771 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:00:08.923129 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:08.923497 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:00:08.923845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:06.520063 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:00:08.925006 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:08.925442 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:08.925686 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:06.520063] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:00:08.926283 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Weigher CapacityWeigher returned, weigher value is {max: 390.8, min: 390.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:00:08.926617 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:00:08.934023 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9b6260f7-20eb-49bb-8602-4aaaa7d8823f tempest-VolumesDeleteCascade-379077449 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:08.933516 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:00:10.930116 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Flow 'volume_create_scheduler' (6a101542-53a7-4a11-a380-148f765bb361) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:10.939590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3fcd79ce-b5ec-4fce-9ffb-bda9a7f7f4f1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:10.954887 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3fcd79ce-b5ec-4fce-9ffb-bda9a7f7f4f1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e2e692fe-95a8-4bc3-ad69-91e3949de763),volume_id=e2e692fe-95a8-4bc3-ad69-91e3949de763,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:10.960563 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4dab5e66-6910-4c9f-bd14-9f9f04ee437c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:10.985956 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:00:11.020478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 0, 2, 752718)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:00:11.021217 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:00:11.022618 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:11.023166 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e2e692fe-95a8-4bc3-ad69-91e3949de763) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:00:11.023702 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:08.933516 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:00:11.025629 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:11.027604 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:00:11.029105 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:00:11.030978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:08.933516] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:00:11.032547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Weigher CapacityWeigher returned, weigher value is {max: 389.8, min: 389.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:00:11.034235 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:00:11.034235 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:00:11.034235 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.280000000000001, total_capacity_gb: 19.84,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:11.029860 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:00:11.084270 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4dab5e66-6910-4c9f-bd14-9f9f04ee437c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:11.089109 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-375d58e0-474c-49dc-9ef1-7015fb9f0bab tempest-TaggedBootDevicesTest-1735451821 None] Flow 'volume_create_scheduler' (6a101542-53a7-4a11-a380-148f765bb361) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:14.052032 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-483f81df-b332-4d11-8193-b06e9cb60f19 tempest-VolumesDeleteCascade-379077449 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:00:14.941544 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Flow 'volume_create_scheduler' (474a1b37-669a-4af4-8559-1240ebef08ed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:14.950205 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ee13913e-bcc9-49dd-be42-60376a7a64af) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:14.965793 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ee13913e-bcc9-49dd-be42-60376a7a64af) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(864356f0-cae5-4433-8f7d-166f6228c9f6),volume_id=864356f0-cae5-4433-8f7d-166f6228c9f6,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:14.972653 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c8428e66-2e98-4999-8cf8-c9118ff4ec01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:14.993814 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:00:15.052599 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 0, 14, 46539)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:00:15.058292 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:00:15.058737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:15.066426 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (864356f0-cae5-4433-8f7d-166f6228c9f6) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:00:15.067020 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:14.046539 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:00:15.068460 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:15.076494 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:15.078513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.28, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:14.046539] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:00:15.082711 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:00:15.083502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:00:15.084144 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:00:15.084689 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:15.084331 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:00:15.149975 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c8428e66-2e98-4999-8cf8-c9118ff4ec01) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:15.152498 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1efbd892-ad4f-4c99-ae4c-f7f8746ce470 tempest-TaggedBootDevicesTest-1735451821 None] Flow 'volume_create_scheduler' (474a1b37-669a-4af4-8559-1240ebef08ed) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:16.777251 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Flow 'volume_create_scheduler' (78f8b3ee-df25-4ad2-afbb-77e87228dea6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:16.783920 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (d2f6baf8-5e57-4105-a442-95e035a858df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:16.795145 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (d2f6baf8-5e57-4105-a442-95e035a858df) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d49f5634-0b85-4e4f-8945-d02c42b35a9f),volume_id=d49f5634-0b85-4e4f-8945-d02c42b35a9f,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:16.798856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (02bd530c-a465-4fa0-b736-3d6cdf0913dd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:00:16.820485 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:00:16.860375 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 0, 14, 46539)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:00:16.861086 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:00:16.861638 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:16.862222 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d49f5634-0b85-4e4f-8945-d02c42b35a9f) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:00:16.862830 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:15.084331 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:16.863675 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 325.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:00:16.865207 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:00:16.867259 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:00:16.870766 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:00:16.874563 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:00:16.874563 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:00:16.874563 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:00:16.874563 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:00:16.878945 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:16.880434 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:00:16.881792 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:15.084331] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:00:16.882666 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Weigher CapacityWeigher returned, weigher value is {max: 390.8, min: 390.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:00:16.884986 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:00:16.889568 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:00:16.890263 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.84,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:00:16.889853 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:00:16.956845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (02bd530c-a465-4fa0-b736-3d6cdf0913dd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:00:16.960360 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c73f0f97-5c87-4e57-98b6-8c2abab4752e tempest-TaggedBootDevicesTest-1735451821 None] Flow 'volume_create_scheduler' (78f8b3ee-df25-4ad2-afbb-77e87228dea6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:00:26.524623 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-59e5bcd8-ad16-48a6-b41f-f4c4971c4068 tempest-TaggedBootDevicesTest_v242-1749553438 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:00:27.671442 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-af77edf6-4221-4d8a-93f7-052f7c984107 tempest-TaggedBootDevicesTest_v242-1749553438 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:00:30.048796 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4eff1be7-ff14-4acc-bcc3-ef8fb6b9dcae tempest-TaggedBootDevicesTest_v242-1749553438 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:00:54.569785 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3eb7c19a-8cb2-4453-a38a-421b5ebac0c0 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:01:02.711910 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a2439d9-37f0-4350-bbd5-2d4bf247424a None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:01:54.573635 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8fc62b9c-33fc-4434-9538-cbbd20904bed None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:02:02.723041 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f35a58fd-ba4a-4e8a-91b1-d51e80038890 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:02:38.679848 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Flow 'volume_create_scheduler' (6068d314-029b-43ff-b171-0d227d617b9f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:02:38.686731 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8b631854-c4bd-4b62-ab99-a9948c95c509) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:02:38.692860 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8b631854-c4bd-4b62-ab99-a9948c95c509) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(add4e032-5ff8-4fac-9d73-a3eb8ba1ecfd),volume_id=add4e032-5ff8-4fac-9d73-a3eb8ba1ecfd,volume_properties=VolumeProperties,volume_type=VolumeType(c35b2163-2d4b-4a3b-92a0-1f7880eb8adf))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:02:38.700570 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cefe238d-6896-4d94-a93e-8e3341c25c1e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:02:38.720547 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:02:38.744650 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 2, 2, 709948)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:02:38.747254 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:02:38.748287 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:38.748652 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (add4e032-5ff8-4fac-9d73-a3eb8ba1ecfd) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:02:38.749052 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.37, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:02.709948 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:02:38.750158 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:02:38.752351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:02:38.753835 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:02:38.757932 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:02:38.764188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:38.764188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:38.764188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.37, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:02.709948] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:02:38.764188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Weigher CapacityWeigher returned, weigher value is {max: 392.8, min: 392.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:02:38.764743 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:02:38.764743 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:02:38.764743 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.37, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:38.752457 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:02:38.793549 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cefe238d-6896-4d94-a93e-8e3341c25c1e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:02:38.795988 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-e4a308b4-3819-4f73-b7d4-b39ac4a16b97 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Flow 'volume_create_scheduler' (6068d314-029b-43ff-b171-0d227d617b9f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:02:40.434677 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 2, 2, 709948)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:02:40.437363 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:02:40.438120 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:40.438748 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:40.439776 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:40.440159 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.37, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:38.752457] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:02:40.442638 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:02:40.443243 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:02:40.773078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-052f0716-cc26-431a-aeae-d317260021f7 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:02:43.131099 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f0879618-21f3-4756-ae20-b89f028a03f1 tempest-VolumeRetypeWithoutMigrationTest-1764270062 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:02:49.973412 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Flow 'volume_create_scheduler' (8ec94761-83b0-4a03-bcf1-10d475aaa5ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:02:49.977935 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (577244dc-8a42-4bc6-ba7e-06600918a37e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:02:49.982315 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (577244dc-8a42-4bc6-ba7e-06600918a37e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(282a7e85-d0af-477f-90ae-39809e3c5641),volume_id=282a7e85-d0af-477f-90ae-39809e3c5641,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:02:49.986078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ec1820df-6ad4-4640-9918-769bc455a9e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:02:49.996548 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:02:50.016328 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 2, 43, 123310)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:02:50.017115 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:02:50.017847 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:50.018557 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (282a7e85-d0af-477f-90ae-39809e3c5641) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:02:50.019123 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.37, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:43.123310 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:02:50.030793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 387.40000000000003, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:02:50.032617 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:02:50.036549 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:02:50.047076 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:02:50.051495 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:02:50.051495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:50.051495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:50.051495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.37, total_capacity_gb: 19.84,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:43.123310] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:02:50.051495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Weigher CapacityWeigher returned, weigher value is {max: 392.8, min: 392.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:02:50.052377 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:02:50.052377 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:02:50.052377 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.37, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:50.033492 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:02:50.065503 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ec1820df-6ad4-4640-9918-769bc455a9e1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:02:50.067661 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6e755a9c-e23b-4852-8f87-1553ae141f70 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Flow 'volume_create_scheduler' (8ec94761-83b0-4a03-bcf1-10d475aaa5ae) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:02:51.262732 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.37, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 2, 43, 123310)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:02:51.263959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:02:51.264551 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:51.265109 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:02:51.265877 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.37, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:50.033492 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:02:51.268148 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:51.268916 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:02:51.269424 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.37, total_capacity_gb: 19.84,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:50.033492] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:02:51.269940 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:02:51.270374 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:02:51.271133 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-14b8b040-89cb-4fff-a9fc-ec5934bfb3ca tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.37, total_capacity_gb: 19.84,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:02:51.270764 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:02:54.582067 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-acb20691-8d4c-4499-8f01-4a272f70fb07 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:02:55.925517 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-32c58e51-d09b-4031-bba8-fd918aae6c79 tempest-VolumeSnapshotQuotasNegativeTestJSON-948497645 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.84, 'free_capacity_gb': 19.34, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:02.705886 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8fab73c5-6b9f-4e0e-a73e-7a5debfd7eb7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.83, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:02.739170 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Flow 'volume_create_scheduler' (85dac276-53d7-487e-845f-eea4c3fd1178) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:02.745583 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4f53fb32-b496-43b7-8eeb-60c9b197dfa2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:02.752332 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4f53fb32-b496-43b7-8eeb-60c9b197dfa2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(74391ca2-fdb2-41a9-a2e7-4ba923eb11bf),volume_id=74391ca2-fdb2-41a9-a2e7-4ba923eb11bf,volume_properties=VolumeProperties,volume_type=VolumeType(fcf0d964-6182-4e0f-95b6-4aaddc62329d))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:02.757348 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a67df14f-23f6-449a-8fc9-de752e96ad23) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:02.778341 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:02.796707 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.83, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 2, 680540)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:02.797389 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:02.797762 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:02.798207 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (74391ca2-fdb2-41a9-a2e7-4ba923eb11bf) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:02.798640 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.83,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:02.680540 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 387.40000000000003, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:02.802557 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:03:02.804553 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 367.40000000000003, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:02.806948 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:02.811824 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:02.820794 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:02.820794 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:02.820794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:02.820794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:02.820794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.83,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:02.680540] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:02.820794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Weigher CapacityWeigher returned, weigher value is {max: 392.59999999999997, min: 392.59999999999997} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:02.821813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:02.821813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:02.821813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.83,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:02.815216 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:02.866637 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a67df14f-23f6-449a-8fc9-de752e96ad23) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:02.870300 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-33416923-247b-4937-9365-e137903e08af tempest-VolumeTypesAccessTest-33218179 None] Flow 'volume_create_scheduler' (85dac276-53d7-487e-845f-eea4c3fd1178) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:05.453829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0d7213b2-d325-41ba-b003-db7c2ae19a95 tempest-VolumeTypesAccessTest-33218179 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.83, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:17.878545 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Flow 'volume_create_scheduler' (2a26d1f2-b5c8-4b77-87e4-90af657fbf7b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:17.881453 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f5954233-98a2-4d1f-b335-b9b9fb45d4bb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:17.884997 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f5954233-98a2-4d1f-b335-b9b9fb45d4bb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b97bb1b8-981c-4ef9-bba7-5746035b7c09),volume_id=b97bb1b8-981c-4ef9-bba7-5746035b7c09,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:17.888319 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5e9fcbd3-618d-410e-9d8d-a6243909b038) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:17.897137 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:17.925999 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.83, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 5, 446806)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:17.926584 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:17.927097 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:17.927643 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b97bb1b8-981c-4ef9-bba7-5746035b7c09) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:17.928129 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.83,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:05.446806 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:17.929884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:17.932563 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:17.935392 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:17.937234 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:17.942249 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:17.942249 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:17.942972 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:17.943783 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.83,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:05.446806] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:17.944506 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Weigher CapacityWeigher returned, weigher value is {max: 392.59999999999997, min: 392.59999999999997} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:17.945290 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:17.946015 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:17.947153 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.83,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:17.946769 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:17.992062 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5e9fcbd3-618d-410e-9d8d-a6243909b038) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:17.994272 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-7a15f8c5-d57d-4d30-81c7-8016637463d9 req-b4cfb96a-0619-45ce-bcc9-4d793661e203 tempest-VolumesGetTestJSON-117686070 None] Flow 'volume_create_scheduler' (2a26d1f2-b5c8-4b77-87e4-90af657fbf7b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:20.702160 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-d45e5249-6a0d-4599-9253-d4bdbb071df2 req-faa7227c-846e-45a4-a216-8ed80b5d6e05 tempest-VolumesGetTestJSON-117686070 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:32.630093 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (bad37ff6-73e2-4210-950c-193e933509ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:32.668075 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c4befd66-5652-49e4-b896-b35383aa694a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:32.677029 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c4befd66-5652-49e4-b896-b35383aa694a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(51e81c99-752e-42ca-8162-1543dc02f25b),volume_id=51e81c99-752e-42ca-8162-1543dc02f25b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:32.679899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d518ba61-c232-4748-b0a9-5a1d65923367) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:32.692311 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:32.714900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 20, 693177)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:32.715686 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:32.716436 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:32.717869 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (51e81c99-752e-42ca-8162-1543dc02f25b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:32.718651 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.82,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:20.693177 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:32.721266 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:32.721266 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:32.721266 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.82,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:20.693177] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:32.723062 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Weigher CapacityWeigher returned, weigher value is {max: 392.4, min: 392.4} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:32.723639 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:32.724414 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:32.725441 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:32.724654 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:32.831842 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d518ba61-c232-4748-b0a9-5a1d65923367) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:32.834973 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-88e4ab5d-2dbb-4deb-934a-70d898887b8e tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (bad37ff6-73e2-4210-950c-193e933509ae) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:42.770966 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (a820052a-4b9b-42b4-af9d-1e104fcf8db8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:42.775435 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0518861f-f24e-48d2-92c9-73fad6656f55) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:42.778626 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0518861f-f24e-48d2-92c9-73fad6656f55) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(df3da152-81ae-41fe-a8ba-7ed8d8029ba3),volume_id=df3da152-81ae-41fe-a8ba-7ed8d8029ba3,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:42.780906 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3141b908-e2cd-43f8-b17e-8a736d017802) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:42.788570 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:42.808060 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 20, 693177)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:42.808706 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:42.809288 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:42.809850 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (df3da152-81ae-41fe-a8ba-7ed8d8029ba3) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:42.810469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:32.724654 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:42.811724 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:42.813849 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:42.815849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:42.817450 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:42.820830 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:42.820830 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:42.822482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:42.823097 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:42.823916 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:32.724654] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:42.824645 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:42.825185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:42.825860 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:42.826300 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.82,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:42.825888 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:42.855323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3141b908-e2cd-43f8-b17e-8a736d017802) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:42.857525 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-80370489-6e14-41b6-ac9a-406b642006c8 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (a820052a-4b9b-42b4-af9d-1e104fcf8db8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:45.001779 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-59b7c8d3-863d-4966-9f00-456cbabf3f47 tempest-VolumesActionsTest-223364317 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:45.644834 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (ff56b0fd-f680-4355-9056-f636df54d13f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:45.648659 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20c5456d-dfde-43db-ae0e-511c6e15c669) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:45.655416 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20c5456d-dfde-43db-ae0e-511c6e15c669) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5621f78c-2935-4d5e-95a3-54424bf1419c),volume_id=5621f78c-2935-4d5e-95a3-54424bf1419c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:45.657399 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (51aaf6ee-fb23-40cc-9f5b-68b305394df7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:45.665713 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:45.682650 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 44, 997459)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:45.683378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:45.683943 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:45.684476 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5621f78c-2935-4d5e-95a3-54424bf1419c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:45.685097 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:44.997459 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:45.686513 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:45.688891 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:45.691882 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:45.694775 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:45.699795 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:45.699795 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:45.699795 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:44.997459] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:45.699795 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:45.700840 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:45.701617 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:45.702549 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.82,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:45.701838 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:45.741943 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (51aaf6ee-fb23-40cc-9f5b-68b305394df7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:45.744831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-480e81bc-d06a-4bfa-bcbf-f4d3e7c4f247 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (ff56b0fd-f680-4355-9056-f636df54d13f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:47.845410 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-baac467a-f70d-4a5e-884f-923660ddc3d5 tempest-VolumesActionsTest-223364317 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:48.805188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (a2c9bf55-7ad4-4af9-9202-789b2bcafcbc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:48.811891 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (26b9af0d-511d-4cde-8a8b-76597909e3c3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:48.815900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (26b9af0d-511d-4cde-8a8b-76597909e3c3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(41dd1d28-e2a2-4d2c-bc49-d623d380851b),volume_id=41dd1d28-e2a2-4d2c-bc49-d623d380851b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:48.818883 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0f02c9f5-16b4-4718-b320-c50bfbcc80a6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:48.838603 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:48.883867 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 47, 840452)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:48.886358 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:48.886917 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:48.887281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (41dd1d28-e2a2-4d2c-bc49-d623d380851b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:48.888277 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:47.840452 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:48.890731 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:48.892884 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:48.895329 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:48.897309 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:48.903095 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:48.903095 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:48.903095 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.82,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:47.840452] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:48.903973 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:48.905135 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:48.905794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:48.906377 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.82,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:48.905982 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:48.950409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0f02c9f5-16b4-4718-b320-c50bfbcc80a6) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:48.954406 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-41478ca8-7ec6-4e1c-b351-85cac6a0aec7 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (a2c9bf55-7ad4-4af9-9202-789b2bcafcbc) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:50.987948 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 47, 840452)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:50.992569 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:50.997486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:50.998096 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:50.998571 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:50.999049 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:50.999555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.82,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:48.905982] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:03:51.001019 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:51.004662 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:51.031845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.82, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 47, 840452)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:51.032631 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:51.033964 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:51.034629 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:51.035946 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:51.039393 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:51.040992 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.82,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:48.905982] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:03:51.041829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:51.043766 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:51.044292 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-921c6faa-5a17-4eb8-96db-19716af2a6b8 tempest-GroupsTest-323837157 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:03:51.452054 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0fb30307-3cbf-41a9-8a11-b9b90406be79 tempest-VolumesActionsTest-223364317 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:51.984616 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (367d825a-ebec-4791-827c-e89a059fd78d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:51.987180 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7e6bb13e-a73e-4840-ae1e-c1ed661e338b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:51.991021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7e6bb13e-a73e-4840-ae1e-c1ed661e338b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(42d9de4b-9270-45dd-8803-58abd2ab1246),volume_id=42d9de4b-9270-45dd-8803-58abd2ab1246,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:51.995222 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (43e03319-a9d5-479c-ab08-6e4245918045) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:52.005374 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:52.026838 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 51, 444665)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:52.027264 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:52.027718 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.028088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (42d9de4b-9270-45dd-8803-58abd2ab1246) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:52.028489 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:51.444665 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:52.031023 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.031424 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.031831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:51.444665] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:52.032576 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Weigher CapacityWeigher returned, weigher value is {max: 391.2, min: 391.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:52.033189 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:52.033722 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:52.034148 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:52.033698 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:52.088746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (43e03319-a9d5-479c-ab08-6e4245918045) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:52.115485 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0584c131-d6ec-49d9-a6ff-910e4ab1d3c6 tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (367d825a-ebec-4791-827c-e89a059fd78d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:52.414604 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 51, 444665)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:52.438175 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:52.438778 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.440054 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:52.440567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.444608 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.446477 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:52.033698] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:03:52.447666 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 390.2, min: 390.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:52.448977 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:52.464899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 51, 444665)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:52.465295 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:52.465659 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.466011 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:52.466387 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.466736 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:52.467100 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:52.033698] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:03:52.469039 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 390.2, min: 390.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:52.473363 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:52.476173 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a8ab433d-aba1-455c-a1b2-4e84631c69a4 tempest-GroupsTest-323837157 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:03:54.158593 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a878d65b-4c71-476b-abfc-80de75c42497 tempest-VolumesActionsTest-223364317 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:54.216913 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Flow 'volume_create_scheduler' (38d79cc3-944a-4d0e-a9ef-f5026334605e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:54.223750 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c8b8fd69-6f28-44ed-81ec-e057e3869b80) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:54.227675 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c8b8fd69-6f28-44ed-81ec-e057e3869b80) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=7fd51ebf-ad47-4645-8d4a-741645432413,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ff08db39-cb8f-4c91-b1d4-b7604f577c62),volume_id=ff08db39-cb8f-4c91-b1d4-b7604f577c62,volume_properties=VolumeProperties,volume_type=VolumeType(194ba21a-1c75-4254-8061-088d943c7ca4))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:54.242346 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9e89a47c-382b-4c6f-9ce2-428a56a11158) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:54.261481 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:54.294655 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 54, 129456)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:54.295065 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:54.295625 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:54.296143 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ff08db39-cb8f-4c91-b1d4-b7604f577c62) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:54.296707 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:54.129456 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:54.298156 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:54.300730 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:54.303839 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:54.310306 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:54.314796 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:54.314796 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:54.314796 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:54.314796 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:54.314796 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:54.129456] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:54.314796 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 391.2, min: 391.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:54.315611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:54.315611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:54.315611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:54.309316 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:54.381555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9e89a47c-382b-4c6f-9ce2-428a56a11158) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:54.392947 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4a17b65d-7bcc-4d0b-8673-3231ae4b9ce4 tempest-GroupsTest-323837157 None] Flow 'volume_create_scheduler' (38d79cc3-944a-4d0e-a9ef-f5026334605e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:54.580013 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dca1c919-5507-4851-b8aa-bf7fda912bba None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:03:55.028124 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (4c9bb361-80dd-4d40-94be-e98fb29ff693) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:55.046863 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1c75f22c-b1b2-46b0-bb3c-930cfab7ae02) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:55.062408 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1c75f22c-b1b2-46b0-bb3c-930cfab7ae02) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(be82342b-a517-440e-aaa6-6e364db41bb0),volume_id=be82342b-a517-440e-aaa6-6e364db41bb0,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:55.070235 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (03ef7228-2c46-4d20-9def-c4f83dca6bda) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:03:55.088424 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:03:55.129370 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 54, 129456)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:55.129682 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:55.130091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:55.130502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (be82342b-a517-440e-aaa6-6e364db41bb0) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:55.131037 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:54.309316 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:55.135079 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:03:55.137280 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:03:55.139037 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:03:55.141074 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:03:55.144825 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:03:55.144825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:55.144825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:55.144825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:54.309316] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:03:55.144825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Weigher CapacityWeigher returned, weigher value is {max: 390.2, min: 390.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:55.144825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:55.146932 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:03:55.146932 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.81,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:55.144025 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:03:55.226492 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (03ef7228-2c46-4d20-9def-c4f83dca6bda) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:03:55.233477 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6799f289-4e82-4f4f-8157-a22f3244cb6c tempest-VolumesActionsTest-290021599 None] Flow 'volume_create_scheduler' (4c9bb361-80dd-4d40-94be-e98fb29ff693) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:03:57.306654 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b1622bee-2280-4c6e-ade5-d26bf6cc1fe5 tempest-GroupsTest-323837157 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:58.187971 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e437ac26-b226-4181-90f0-ce44e551f61a tempest-VolumesActionsTest-290021599 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:58.572470 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6e69c1c4-e7a6-4243-90eb-166e11393360 tempest-GroupsTest-323837157 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:59.605143 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d809ab03-7a3a-4729-ac8f-ee1297a82bce tempest-VolumesActionsTest-290021599 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:03:59.766409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 59, 597651)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:59.766902 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:59.767601 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:59.767906 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:59.768621 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:59.769447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:59.770083 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:59.597651] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:03:59.770984 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 391.2, min: 391.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:59.771554 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:59.780147 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 59, 597651)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:03:59.780934 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:03:59.781526 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:59.782132 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:03:59.782704 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:59.783298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:03:59.784100 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:59.597651] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:03:59.785668 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 391.2, min: 391.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:03:59.786307 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:03:59.786888 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1eedf158-f981-475b-92be-f5b84d694bd6 tempest-GroupsTest-323837157 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:04:01.386181 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Flow 'volume_create_scheduler' (c322cbf8-1a13-4f64-807e-c3d810a41687) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:01.389144 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ddd2ef39-8e9d-452f-a3bf-35f08daf6516) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:01.392081 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ddd2ef39-8e9d-452f-a3bf-35f08daf6516) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fc88d47c-2acf-41eb-b115-7838312047bf,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d0214349-0078-48f7-b1f5-0889c1945556),volume_id=d0214349-0078-48f7-b1f5-0889c1945556,volume_properties=VolumeProperties,volume_type=VolumeType(81a6fc6a-7931-431d-b6bd-dce6c7146dd9))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:01.394645 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (dcc652c8-3023-4955-82c6-b40495802952) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:01.403748 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:01.421824 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 3, 59, 597651)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:01.422663 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:01.423325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:01.423911 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d0214349-0078-48f7-b1f5-0889c1945556) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:01.424531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:59.597651 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:01.425841 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:01.428196 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:01.431061 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:01.433187 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:01.438208 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:01.438208 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:01.438208 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:03:59.597651] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:01.438208 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 391.2, min: 391.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:01.438867 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:01.439410 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:01.439962 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:01.439595 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:01.475423 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (dcc652c8-3023-4955-82c6-b40495802952) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:01.477814 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-07dfd210-04eb-4018-8a12-f0bbe2aba02c tempest-GroupsTest-323837157 None] Flow 'volume_create_scheduler' (c322cbf8-1a13-4f64-807e-c3d810a41687) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:02.739837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b083af02-ec08-448f-a006-b746c9cd4c42 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:03.491623 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Flow 'volume_create_scheduler' (0106658f-6ae6-484b-9479-0310e4e8d47b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:03.497305 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (868c0e8d-00cb-4d53-a44c-14592a18027a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:03.520699 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (868c0e8d-00cb-4d53-a44c-14592a18027a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=fc88d47c-2acf-41eb-b115-7838312047bf,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0fd0efed-1260-4c6c-a069-771e81d5dcdd),volume_id=0fd0efed-1260-4c6c-a069-771e81d5dcdd,volume_properties=VolumeProperties,volume_type=VolumeType(81a6fc6a-7931-431d-b6bd-dce6c7146dd9))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:03.554533 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bffbbbb1-b3f4-4ab6-ac62-f0b2a3cf8238) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:03.570965 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:03.590493 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 2, 734490)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:03.591188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:03.591676 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:03.592554 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0fd0efed-1260-4c6c-a069-771e81d5dcdd) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:03.592746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:02.734490 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:03.594564 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:03.595218 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:03.595822 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:02.734490] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:03.596485 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Weigher CapacityWeigher returned, weigher value is {max: 390.2, min: 390.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:03.597089 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:03.597691 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:03.598336 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:03.597947 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:03.645065 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bffbbbb1-b3f4-4ab6-ac62-f0b2a3cf8238) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:03.660111 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f72b1dc9-bb3e-40c3-84ae-989554b95acd tempest-GroupsTest-323837157 None] Flow 'volume_create_scheduler' (0106658f-6ae6-484b-9479-0310e4e8d47b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:08.150387 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Flow 'volume_create_scheduler' (16df605a-9847-4b53-a804-c43efb7cc68f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:08.158513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2f75f1b7-e9c2-4779-ba68-a1919cf16658) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:08.168035 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2f75f1b7-e9c2-4779-ba68-a1919cf16658) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(1f8b5543-4e24-4d5f-bf04-c2f06cbd7dae),volume_id=1f8b5543-4e24-4d5f-bf04-c2f06cbd7dae,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:08.171242 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c8c0da41-0b34-40d0-abd1-d75876b7fff4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:08.187730 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:08.238376 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 2, 734490)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:08.239571 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:08.240203 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:08.240981 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1f8b5543-4e24-4d5f-bf04-c2f06cbd7dae) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:08.241829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:03.597947 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:08.246978 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:08.249367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:08.252877 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:08.262433 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:08.267513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:08.267513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:08.267513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:03.597947] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:08.267513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Weigher CapacityWeigher returned, weigher value is {max: 389.2, min: 389.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:08.267513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:08.268003 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:08.268003 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:08.259974 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:08.340785 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c8c0da41-0b34-40d0-abd1-d75876b7fff4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:08.350632 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c83dbb31-e81d-4006-b97a-ba422937a34d tempest-VolumesListAdminTestJSON-1154432475 None] Flow 'volume_create_scheduler' (16df605a-9847-4b53-a804-c43efb7cc68f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:09.959233 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Flow 'volume_create_scheduler' (e1eb624f-ef4f-415a-96e3-099e726bc059) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:09.969226 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6bf9eadd-2cea-4b61-86ec-7c3b83e76c44) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:09.974037 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6bf9eadd-2cea-4b61-86ec-7c3b83e76c44) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a8f84ad4-84ed-456c-a920-526dd301d1b3),volume_id=a8f84ad4-84ed-456c-a920-526dd301d1b3,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:09.996568 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (08251291-f3f2-4746-8f7a-ccbb48b69635) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:10.013571 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:10.064859 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 2, 734490)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:10.065236 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:10.065605 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:10.065966 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a8f84ad4-84ed-456c-a920-526dd301d1b3) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:10.066395 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:08.259974 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:10.075500 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:10.077772 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:10.079855 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:10.082639 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:10.084763 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:10.084763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:10.084763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:10.084763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:08.259974] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:10.084763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Weigher CapacityWeigher returned, weigher value is {max: 388.2, min: 388.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:10.085351 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:10.085351 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:10.085351 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:10.078407 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:10.127451 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (08251291-f3f2-4746-8f7a-ccbb48b69635) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:10.130194 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-05f6c28c-d62a-4361-9ac2-5e7203f8a76f tempest-VolumesListAdminTestJSON-1154432475 None] Flow 'volume_create_scheduler' (e1eb624f-ef4f-415a-96e3-099e726bc059) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:10.254025 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-802383af-95c4-4430-9aa7-d3148d0e73c2 tempest-GroupsTest-323837157 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:11.567509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Flow 'volume_create_scheduler' (2fbdb11c-e8c2-4bbb-a6d4-40202a3959fe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:11.570914 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (5bc52a71-3764-45ee-acfa-5b1b54a3d79d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:11.574216 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (5bc52a71-3764-45ee-acfa-5b1b54a3d79d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(808f235c-9243-49d4-8b61-cc544d7be532),volume_id=808f235c-9243-49d4-8b61-cc544d7be532,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:11.576279 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e586a9d2-96dd-499a-b9f4-4c77805e944f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:11.585027 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:11.604651 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 10, 247816)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:11.605628 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:11.606246 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:11.606802 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (808f235c-9243-49d4-8b61-cc544d7be532) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:11.607447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:10.247816 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:11.609066 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:11.610880 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:11.613100 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:11.617085 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:11.622837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:11.622837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:11.622837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:10.247816] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:11.622837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Weigher CapacityWeigher returned, weigher value is {max: 388.2, min: 388.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:11.622837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:11.624213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:11.624213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:11.619871 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:11.664752 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e586a9d2-96dd-499a-b9f4-4c77805e944f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:11.666870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7b704f6b-9232-4d5f-b31b-d672ed61a06f tempest-VolumesListAdminTestJSON-1154432475 None] Flow 'volume_create_scheduler' (2fbdb11c-e8c2-4bbb-a6d4-40202a3959fe) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:13.579685 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Flow 'volume_create_scheduler' (89e1e2da-2c33-4a15-b248-57778a851e1b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:13.587321 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8fbbcad1-d835-45a7-9e70-1b26e81df65b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:13.594365 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8fbbcad1-d835-45a7-9e70-1b26e81df65b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fdf60422-7009-4e8a-9392-a227b5a54794),volume_id=fdf60422-7009-4e8a-9392-a227b5a54794,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:13.599810 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2a55e6f-9dc0-45c8-8d5a-d85bcdbdccbe) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:13.610680 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:13.637778 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 10, 247816)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:13.637778 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:13.637778 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:13.637778 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (fdf60422-7009-4e8a-9392-a227b5a54794) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:13.638219 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:11.619871 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:13.638219 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:13.638219 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:13.638219 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:11.619871] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:13.638547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Weigher CapacityWeigher returned, weigher value is {max: 387.2, min: 387.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:13.638631 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:13.639891 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:13.639891 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.81,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:13.639112 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:13.764599 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2a55e6f-9dc0-45c8-8d5a-d85bcdbdccbe) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:13.767662 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b2c9a607-9496-4399-99df-c17e26469ab0 tempest-VolumesListAdminTestJSON-1488893033 None] Flow 'volume_create_scheduler' (89e1e2da-2c33-4a15-b248-57778a851e1b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:15.781028 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d88148f7-103e-4754-9fc1-165aa6650a40 tempest-VolumesListAdminTestJSON-1488893033 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:16.015171 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4ea4587c-32c9-4c9f-9423-64f34ff26416 tempest-VolumesListAdminTestJSON-1154432475 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:17.187211 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 16, 7752)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:17.189004 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:17.190302 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:17.192521 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:17.193102 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:17.193803 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:17.194387 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:16.007752] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:04:17.198043 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Weigher CapacityWeigher returned, weigher value is {max: 387.2, min: 387.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:17.199932 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:17.219746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 16, 7752)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:17.223064 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:17.223804 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:17.224339 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:17.225017 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:17.226607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:17.229032 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:16.007752] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:04:17.234995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Weigher CapacityWeigher returned, weigher value is {max: 387.2, min: 387.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:17.234995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:17.234995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ef02bb97-7f34-495c-a0c5-da68ce82ac09 tempest-GroupsV320Test-190206346 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:04:17.252701 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ebfd8704-8443-4a24-9940-d5d0bd2b7488 tempest-VolumesListAdminTestJSON-1154432475 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:18.193667 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-902f9a7b-e1ff-49ab-a2f0-3af4673dbd98 tempest-VolumesListAdminTestJSON-1154432475 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:19.086036 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3e70a724-0d73-4c31-9faa-71ee8e4be7f0 tempest-GroupsV320Test-190206346 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:25.118146 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Flow 'volume_create_scheduler' (03108c07-d9e9-49d1-81df-59c9746a57ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:25.121983 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e50fbe0a-2465-4cd8-a9d0-67106569dcab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:25.130306 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e50fbe0a-2465-4cd8-a9d0-67106569dcab) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9fb31c3c-dded-4334-ad39-74f975cfe4ba),volume_id=9fb31c3c-dded-4334-ad39-74f975cfe4ba,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:25.130306 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ee1154a6-cb12-42d6-9000-d99254c3e5be) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:25.138482 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:25.154718 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 19, 81787)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:25.155279 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:25.155743 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:25.156305 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9fb31c3c-dded-4334-ad39-74f975cfe4ba) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:25.156926 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:19.081787 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:25.158431 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:25.161188 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:25.167793 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:25.170177 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:25.172733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:25.172733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:25.172733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.81,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:19.081787] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:25.172733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Weigher CapacityWeigher returned, weigher value is {max: 389.2, min: 389.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:25.172733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:25.173214 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:25.173214 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:25.163647 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:25.201164 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ee1154a6-cb12-42d6-9000-d99254c3e5be) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:25.203269 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b7f00cbd-2c23-425a-bdb2-786aa663d60a tempest-VolumesImageMetadata-3851237 None] Flow 'volume_create_scheduler' (03108c07-d9e9-49d1-81df-59c9746a57ae) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:25.563256 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Flow 'volume_create_scheduler' (55ad6df8-cadd-4c33-bfbc-f194a5b695af) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:25.566235 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (33344fa1-38b9-4a45-8f04-2003b88d7130) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:25.569165 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (33344fa1-38b9-4a45-8f04-2003b88d7130) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e429effe-6f3c-4293-bee0-0e0626b8042a),volume_id=e429effe-6f3c-4293-bee0-0e0626b8042a,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:25.571482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (112add97-48a7-4128-a457-b0794fdca821) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:25.579425 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:25.597885 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 19, 81787)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:25.598522 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:25.598955 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:25.599463 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e429effe-6f3c-4293-bee0-0e0626b8042a) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:25.599870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:25.163647 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:25.600914 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:25.604237 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:25.606085 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:25.610323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:25.613925 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:25.613925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:25.613925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:25.613925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.26, total_capacity_gb: 19.81,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:25.163647] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:25.613925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Weigher CapacityWeigher returned, weigher value is {max: 388.2, min: 388.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:25.614671 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:25.614671 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:25.614671 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.81,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:25.603901 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:25.634748 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (112add97-48a7-4128-a457-b0794fdca821) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:25.637269 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0089f42b-30e5-41e3-ac9e-da392e02accb tempest-VolumesServicesTestJSON-1154624079 None] Flow 'volume_create_scheduler' (55ad6df8-cadd-4c33-bfbc-f194a5b695af) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:27.321013 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fa0e4c01-a07a-4289-ab32-445e34ce93fd tempest-VolumesServicesTestJSON-1154624079 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:29.049371 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-079b3320-576c-4d9c-beeb-4df971c67cdf tempest-VolumesImageMetadata-3851237 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:32.245594 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (23a4652c-920c-4755-99ec-7d5ee23cae81) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:32.251202 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (48e522ee-ebd9-4db4-a688-1e3bbacb6ac7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:32.256708 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (48e522ee-ebd9-4db4-a688-1e3bbacb6ac7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e4b75b6e-8e31-43dd-8e1d-2c8fccea0584),volume_id=e4b75b6e-8e31-43dd-8e1d-2c8fccea0584,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:32.259477 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1fd86611-eba6-4feb-85df-ff2f9b00bd12) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:32.266783 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:32.285402 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 29, 43303)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:32.286072 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:32.286614 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:32.286970 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e4b75b6e-8e31-43dd-8e1d-2c8fccea0584) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:32.287533 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:29.043303 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:32.288775 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.20000000000005, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:32.291308 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:32.293795 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:32.296105 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:32.302357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:32.302357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:32.302357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:29.043303] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:32.302357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:32.302357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:32.302798 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:32.302798 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:32.300895 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:32.331027 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1fd86611-eba6-4feb-85df-ff2f9b00bd12) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:32.336040 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-49b35d84-0e1c-4075-8e37-5b0416b94b6b tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (23a4652c-920c-4755-99ec-7d5ee23cae81) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:33.834683 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (eb33afaa-8c45-47c9-b8df-bfdb0e6b8489) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:33.844890 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4288a815-ff8f-4ca3-a9ff-a4202a5e0622) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:33.847374 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4288a815-ff8f-4ca3-a9ff-a4202a5e0622) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(3414da0a-e428-4688-a451-ab8a3928aeea),volume_id=3414da0a-e428-4688-a451-ab8a3928aeea,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:33.854967 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ed6520dd-70bb-4f00-9285-ea158232cdba) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:33.864585 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:33.894793 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 29, 43303)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:33.896243 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:33.897982 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:33.899753 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (3414da0a-e428-4688-a451-ab8a3928aeea) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:33.900327 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:32.300895 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:33.903296 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:33.904315 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:33.905169 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:32.300895] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:33.906119 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:33.906899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:33.907774 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:33.908381 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:33.907782 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:33.948893 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ed6520dd-70bb-4f00-9285-ea158232cdba) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:33.951209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9de7d84b-8b00-446d-987c-6743185ca69e tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (eb33afaa-8c45-47c9-b8df-bfdb0e6b8489) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:35.342592 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Flow 'volume_create_scheduler' (9d750baa-1590-4e88-bdc5-7380dda22a0d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:35.348180 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cc37c00e-4f78-4fa1-b963-2082ec74db2a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:35.352964 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cc37c00e-4f78-4fa1-b963-2082ec74db2a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dee4c3f8-86cc-4b43-ae79-46ad0eda1497),volume_id=dee4c3f8-86cc-4b43-ae79-46ad0eda1497,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:35.356846 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (787129c6-6703-4d54-9f39-030d61b44577) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:35.365457 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:35.398298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 29, 43303)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:35.398298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:35.398298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:35.398298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (dee4c3f8-86cc-4b43-ae79-46ad0eda1497) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:35.400233 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:33.907782 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:35.401460 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:35.404202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:35.406321 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:35.408523 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:35.412812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:35.412812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:35.412812 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:33.907782] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:35.413611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:35.414843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:35.415569 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:35.416148 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:35.415776 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:35.447722 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (787129c6-6703-4d54-9f39-030d61b44577) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:35.450378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9533afac-a323-46cb-861a-a8c48d57aa09 tempest-VolumesTransfersV355Test-2071470822 None] Flow 'volume_create_scheduler' (9d750baa-1590-4e88-bdc5-7380dda22a0d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:35.530468 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-26602a16-f431-41bc-8bf7-02d0ef429cc6 tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:36.667721 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ccdd476e-e60a-4ed0-b436-af59e128776f tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:37.496681 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (cd4e92a6-c95e-4592-b98e-ea62750880d0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:37.500730 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f58f5a18-0e61-4a15-9885-ab4394f308cb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:37.503427 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f58f5a18-0e61-4a15-9885-ab4394f308cb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f0c6e75c-a9e8-4bb3-94ce-7544904ffe3c),volume_id=f0c6e75c-a9e8-4bb3-94ce-7544904ffe3c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:37.505563 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f0ef24ad-5761-4172-9a57-1915ac88c864) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:37.523182 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:37.552393 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 36, 662159)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:37.553363 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:37.553955 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:37.554517 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f0c6e75c-a9e8-4bb3-94ce-7544904ffe3c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:37.555267 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:36.662159 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:37.556722 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:37.558996 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:37.561111 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:37.563737 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:37.565899 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:37.568800 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:37.568800 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:37.568800 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:36.662159] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:37.568800 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:37.568800 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:37.569740 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:37.571000 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:37.570023 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:37.602350 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f0ef24ad-5761-4172-9a57-1915ac88c864) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:37.605130 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-6165b6df-97fe-4719-b888-dfc7ae0471ec tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (cd4e92a6-c95e-4592-b98e-ea62750880d0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:38.301154 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8b2d88b-68ba-40cf-bce4-0b7d7bb4f0ac tempest-VolumesTransfersV355Test-324380987 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:38.894088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (f328b5f9-93bc-42ad-a11c-d4f7ac500317) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:38.896758 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (02f55ce6-e47f-4591-89ac-16e3469f4a67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:38.900018 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (02f55ce6-e47f-4591-89ac-16e3469f4a67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=None,source_replicaid=,source_volid=f0c6e75c-a9e8-4bb3-94ce-7544904ffe3c,volume=Volume(5c52a736-2fc8-49a4-9719-92c27fae304b),volume_id=5c52a736-2fc8-49a4-9719-92c27fae304b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:38.902420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (23ee582a-1816-4182-acad-8ab7479707ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:38.909934 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:38.925410 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 38, 292165)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:38.926152 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:38.926678 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:38.927395 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5c52a736-2fc8-49a4-9719-92c27fae304b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:38.927795 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:38.292165 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:38.928850 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:38.931105 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:38.933539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:38.935724 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:38.937900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:38.937900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:38.937900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:38.292165] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:38.937900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:38.937900 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:38.938339 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:38.938339 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:38.931248 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:38.963971 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (23ee582a-1816-4182-acad-8ab7479707ab) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:38.966514 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3bf49a71-1330-4ff7-aeae-53611a270fc8 tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (f328b5f9-93bc-42ad-a11c-d4f7ac500317) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:39.120970 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Flow 'volume_create_scheduler' (0f4ee2dc-4cde-431a-9577-c0112917a3b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:39.123494 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c91fd86c-2450-467f-b30b-16680d2737d9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:39.132236 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c91fd86c-2450-467f-b30b-16680d2737d9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e5791d2e-003b-46fd-9818-4ec171cc4c5b),volume_id=e5791d2e-003b-46fd-9818-4ec171cc4c5b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:39.136519 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f7be3650-63ad-4120-a8ce-21aaf341ff30) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:39.144086 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:39.166112 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 38, 292165)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:39.166777 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:39.167302 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:39.167882 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e5791d2e-003b-46fd-9818-4ec171cc4c5b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:39.169146 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:38.931248 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:39.171389 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:39.172052 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:39.172614 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:38.931248] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:39.173567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:39.174154 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:39.174813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:39.175397 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:39.175047 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:39.208840 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f7be3650-63ad-4120-a8ce-21aaf341ff30) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:39.212641 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0bd4407f-d9eb-4057-a6ea-be5509ad2511 tempest-VolumesTransfersV355Test-2071470822 None] Flow 'volume_create_scheduler' (0f4ee2dc-4cde-431a-9577-c0112917a3b6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:40.671683 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (12bf85e4-f433-42d5-8e3d-242c646eebfc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:40.674228 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1d4c656c-f603-4057-9bbd-ede18f299551) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:40.676718 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1d4c656c-f603-4057-9bbd-ede18f299551) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7e55576c-eccb-4f12-b32c-16fd86218c5b),volume_id=7e55576c-eccb-4f12-b32c-16fd86218c5b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:40.682938 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (846ad3c7-dd4f-4cf3-864b-7224f3926f22) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:40.701305 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:40.721869 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 38, 292165)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:40.722285 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:40.724270 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:40.724804 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (7e55576c-eccb-4f12-b32c-16fd86218c5b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:40.725346 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:39.175047 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:40.727094 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:40.728590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:40.729790 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:40.731266 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:40.734658 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:40.735298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:40.735850 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:39.175047] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:40.736478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:40.736997 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:40.737943 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:40.738638 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:40.738205 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:40.773090 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (846ad3c7-dd4f-4cf3-864b-7224f3926f22) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:40.776275 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7125c186-fe8c-4df0-b102-2163922a1f10 tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (12bf85e4-f433-42d5-8e3d-242c646eebfc) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:41.393563 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-619fc900-ae22-49cf-94d0-540b8f61cc2d tempest-VolumesTransfersV355Test-324380987 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:42.500596 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ab483590-661e-4102-9516-26df6fea7134 tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:44.707381 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f6a9d648-cd00-448e-86d3-a3318500641d tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:45.766299 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (2196c71b-403d-4f2b-9736-ca80f087f590) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:45.770497 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (73bab5b2-0334-406d-9b44-36105770a9be) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:45.776430 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (73bab5b2-0334-406d-9b44-36105770a9be) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9b3d8e1c-fd37-46c6-ade4-01597fc05b4a),volume_id=9b3d8e1c-fd37-46c6-ade4-01597fc05b4a,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:45.780200 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8c83dd6b-7550-4e3e-8a87-7ab8cef1d9bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:45.790516 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:45.821033 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 44, 698508)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:45.822232 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:45.822801 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:45.823693 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9b3d8e1c-fd37-46c6-ade4-01597fc05b4a) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:45.824279 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:44.698508 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:45.825771 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:45.827070 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:45.828244 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:45.829631 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:45.830833 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:45.832995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:45.833418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:45.833799 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:44.698508] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:45.834274 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:45.834782 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:45.835620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:45.836038 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:45.835646 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:45.888565 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8c83dd6b-7550-4e3e-8a87-7ab8cef1d9bf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:45.891479 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-7f1ee14f-d531-4eaf-9c67-ebbc723096da tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (2196c71b-403d-4f2b-9736-ca80f087f590) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:46.116747 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Flow 'volume_create_scheduler' (d597368c-9abd-4040-ad2b-14f9c67933d1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:46.127956 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a47103f6-ffae-439e-ace8-564a14e6abab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:46.131217 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a47103f6-ffae-439e-ace8-564a14e6abab) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5f685cab-db4e-400a-bf24-b893891c6221),volume_id=5f685cab-db4e-400a-bf24-b893891c6221,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:46.133826 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (31464bae-7ae6-4ecb-9b59-11bff2280ee2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:46.143257 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:46.161480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 44, 698508)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:46.162388 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:46.162906 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:46.164628 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5f685cab-db4e-400a-bf24-b893891c6221) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:46.165198 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:45.835646 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:46.166784 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:46.168332 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:46.170221 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:46.171784 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:46.173952 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:46.174563 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:46.175081 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:45.835646] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:46.175698 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:46.176204 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:46.177077 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:46.178139 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:46.177226 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:46.216461 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (31464bae-7ae6-4ecb-9b59-11bff2280ee2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:46.218715 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-45c36a1e-a8cd-4a46-af83-f480e153d32c tempest-VolumesExtendTest-618406405 None] Flow 'volume_create_scheduler' (d597368c-9abd-4040-ad2b-14f9c67933d1) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:47.526658 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (c893bd07-31a6-4314-9f4e-26f9d276b87d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:47.534439 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (d97c0d7a-d612-4c95-974e-3e15df066346) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:47.537307 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (d97c0d7a-d612-4c95-974e-3e15df066346) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(676b0e2f-20be-40b4-8e4f-9d92016cd357),volume_id=676b0e2f-20be-40b4-8e4f-9d92016cd357,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:47.547418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b1abc23a-b892-4a08-94db-76a129f3c86b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:47.566301 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:47.599773 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 44, 698508)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:47.600445 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:47.600953 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:47.602184 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (676b0e2f-20be-40b4-8e4f-9d92016cd357) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:47.602732 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:46.177226 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:47.605191 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:47.605741 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:47.606286 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:46.177226] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:47.606859 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:47.607355 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:47.607990 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:47.608509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:47.608155 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:47.650981 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b1abc23a-b892-4a08-94db-76a129f3c86b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:47.655398 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fe237008-4bc0-429f-9805-fc82d5a29bf5 tempest-VolumesGetTest-749057094 None] Flow 'volume_create_scheduler' (c893bd07-31a6-4314-9f4e-26f9d276b87d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:48.511371 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 44, 698508)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:48.518699 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:48.519628 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:48.523999 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Checking if host ncdv-master-1@ceph#ceph can extend the volume 5f685cab-db4e-400a-bf24-b893891c6221 in 1 GB {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:51}} Nov 20 21:04:48.525142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:47.608155 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:48.526109 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:48.528062 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:48.529575 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:48.532759 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:48.539387 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:48.539940 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:48.543853 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:47.608155] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:48.544540 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:48.545040 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:48.545605 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ee31c95a-3af4-4a82-8248-63766083e648 tempest-VolumesExtendTest-618406405 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.25, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:48.545244 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:49.426375 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bc38e633-e248-4671-bc8b-424061dea9a3 tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:50.033055 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Flow 'volume_create_scheduler' (27199d23-a527-4e2e-8731-8d4f28b08063) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:50.042630 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b57e66da-d996-44f1-9ea8-773930d612bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:50.045411 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b57e66da-d996-44f1-9ea8-773930d612bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a2b69ac-7c3f-4d3b-99f1-7ea09bb2992d),volume_id=9a2b69ac-7c3f-4d3b-99f1-7ea09bb2992d,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:50.048014 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (58606393-8fa6-44ad-adba-b9e381ee3abd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:50.057669 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:50.108536 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 49, 421551)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:50.110957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:50.112200 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:50.112653 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9a2b69ac-7c3f-4d3b-99f1-7ea09bb2992d) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:50.113579 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:49.421551 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:50.115497 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:50.120194 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:50.120811 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:49.421551] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:50.121639 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:50.122569 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:50.123197 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:50.124803 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:50.124422 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:50.174346 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (58606393-8fa6-44ad-adba-b9e381ee3abd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:50.176660 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2d14a021-4b35-4160-b087-8daadb74cf7e tempest-VolumesExtendTest-618406405 None] Flow 'volume_create_scheduler' (27199d23-a527-4e2e-8731-8d4f28b08063) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:51.320761 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 49, 421551)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:51.321721 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:51.321824 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:51.322309 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:51.322839 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:50.124422 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 471, in extend_volume Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: backend_state = self.driver.backend_passes_filters( Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 325.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:51.323595 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:51.325111 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:51.327801 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:51.329485 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:51.331697 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:51.331697 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:51.331697 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:51.331697 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:51.331697 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:50.124422] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:51.331697 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Weigher CapacityWeigher returned, weigher value is {max: 384.0, min: 384.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:51.332232 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:51.332232 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-04c1a205-1eef-4e37-a87e-5c310950771e tempest-VolumesExtendTest-618406405 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:51.325833 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:51.933811 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7a57fec8-fb42-4c0f-9e93-0005991b8f22 tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:53.298199 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-877a53b4-e568-4564-85cd-0c105edf0154 tempest-VolumesGetTest-749057094 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:53.707167 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 53, 293543)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:53.711714 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:53.712234 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:53.713047 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Checking if host ncdv-master-1@ceph#ceph can extend the volume 9a2b69ac-7c3f-4d3b-99f1-7ea09bb2992d in 1 GB {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:51}} Nov 20 21:04:53.714703 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:53.293543 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:53.716586 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:53.717627 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:53.720064 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:53.293543] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:53.722185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:53.723147 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:53.724327 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-564163fd-296b-4d82-8403-1d2fb103075c tempest-VolumesExtendTest-618406405 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:53.723187 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:54.574843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dd1556e1-0ae4-406f-accd-566fb1cb03d2 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:04:57.199580 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Flow 'volume_create_scheduler' (b3628246-ac00-4e40-b7d4-4c801927095a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:57.207302 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (91e69583-6d93-4db2-a49e-be75e3b9e706) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:57.213569 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (91e69583-6d93-4db2-a49e-be75e3b9e706) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(6be6eddb-1942-434a-a6ac-c0f87e93ad88),volume_id=6be6eddb-1942-434a-a6ac-c0f87e93ad88,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:57.233642 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c82400a1-cf50-48f0-902f-010f057910c5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:57.256773 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:57.363817 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b7ba527e-634f-4c30-892f-c6f42cab4fc6 tempest-VolumesExtendTest-618406405 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:04:57.368305 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 57, 310106)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:57.369535 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:57.370135 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:57.372424 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (6be6eddb-1942-434a-a6ac-c0f87e93ad88) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:57.373040 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:57.310106 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:57.394585 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:57.397010 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 471, in extend_volume Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: backend_state = self.driver.backend_passes_filters( Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:57.399098 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:57.402133 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:57.406727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:57.406727 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:57.406727 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:57.406727 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:57.406727 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:57.310106] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:57.406727 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:57.407751 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:57.407751 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:57.407751 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:57.405789 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:57.599510 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c82400a1-cf50-48f0-902f-010f057910c5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:57.630755 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ce93590a-d0c1-434c-9db8-c465dd6d6d42 tempest-VolumesListTestJSON-997245241 None] Flow 'volume_create_scheduler' (b3628246-ac00-4e40-b7d4-4c801927095a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:58.872071 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Flow 'volume_create_scheduler' (c9f6f720-8330-43a7-85e3-2597577b6e0f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:58.876439 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (79c35fe7-d599-4db8-90df-cc2d0233b7b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:58.883696 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (79c35fe7-d599-4db8-90df-cc2d0233b7b8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d75e3c80-cf66-440a-b7fd-105abbb9f50a),volume_id=d75e3c80-cf66-440a-b7fd-105abbb9f50a,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:58.887843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8701981d-fd77-4891-823e-4a41db0cf824) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:04:58.908652 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:04:58.957967 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 57, 310106)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:04:58.958782 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:04:58.960297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:58.961349 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d75e3c80-cf66-440a-b7fd-105abbb9f50a) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:04:58.963093 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:57.405789 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:58.967005 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:04:58.968607 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:04:58.970279 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:04:58.973215 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:04:58.977758 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:04:58.977758 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:58.977758 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:04:58.977758 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:57.405789] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:04:58.977758 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:04:58.979051 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:04:58.979051 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:04:58.979051 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:58.970170 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:04:59.043042 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8701981d-fd77-4891-823e-4a41db0cf824) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:04:59.048046 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-a18f3736-8bbd-431b-9cec-0f8edb1ffd7c tempest-VolumesListTestJSON-997245241 None] Flow 'volume_create_scheduler' (c9f6f720-8330-43a7-85e3-2597577b6e0f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:04:59.410252 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e8a2f84a-d695-403a-aff8-1e5fbcfaba89 tempest-VolumesExtendTest-618406405 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:05:00.525618 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Flow 'volume_create_scheduler' (59b0e770-e3a7-4182-a17d-02f8a93175a0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:05:00.529136 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (08cdb3a6-3dac-449d-ae64-5265a2ac25fd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:05:00.537101 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (08cdb3a6-3dac-449d-ae64-5265a2ac25fd) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8f717ef3-4aa5-4897-933e-fb7944998839),volume_id=8f717ef3-4aa5-4897-933e-fb7944998839,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:05:00.541468 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fea475f5-536a-4a0a-899b-03b18cd82ec2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:05:00.558640 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:05:00.579360 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 4, 59, 404931)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:05:00.580103 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:05:00.580639 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:00.581129 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (8f717ef3-4aa5-4897-933e-fb7944998839) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:05:00.581657 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:59.404931 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:05:00.586570 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:05:00.588282 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:05:00.590483 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:05:00.592227 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:05:00.597319 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:00.597745 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:00.598159 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:04:59.404931] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:05:00.598763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:05:00.599265 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:05:00.599913 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:05:00.600447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:00.600086 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:05:00.651055 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fea475f5-536a-4a0a-899b-03b18cd82ec2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:05:00.657630 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c6418434-e2de-4d6f-b591-32d43514e1fb tempest-VolumesListTestJSON-997245241 None] Flow 'volume_create_scheduler' (59b0e770-e3a7-4182-a17d-02f8a93175a0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:05:02.736954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-43476aec-a736-4159-a723-946050a3d442 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.81, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:05:06.439509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-70c0dda3-0b88-450f-9b0d-b5cd956ac788 tempest-VolumesListTestJSON-997245241 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:05:07.831455 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-278bea18-0705-4799-bbe4-a508741883e6 tempest-VolumesListTestJSON-997245241 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:05:08.994592 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-94b8101e-d2d7-4d96-9248-d77ff29e3140 tempest-VolumesListTestJSON-997245241 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:05:14.186991 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Flow 'volume_create_scheduler' (608e39df-1830-466a-a9a0-50e3f9786f56) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:05:14.195345 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4e3671de-033a-4cf4-a2d2-3af25105e47d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:05:14.197956 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4e3671de-033a-4cf4-a2d2-3af25105e47d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dbc31c81-8b0c-4ec0-9814-4ec96849d593),volume_id=dbc31c81-8b0c-4ec0-9814-4ec96849d593,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:05:14.199969 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fc6f4e3d-609c-4006-bb44-13256e6a29d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:05:14.229834 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:05:14.277597 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 5, 8, 985592)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:05:14.278023 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:05:14.278483 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:14.278845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (dbc31c81-8b0c-4ec0-9814-4ec96849d593) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:05:14.291060 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:08.985592 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:05:14.294119 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:14.305379 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:14.305858 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:08.985592] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:05:14.306353 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:05:14.306714 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:05:14.307239 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:05:14.307633 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:14.307266 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:05:14.365313 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fc6f4e3d-609c-4006-bb44-13256e6a29d8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:05:14.374657 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-40281a08-a3d5-47ea-a067-afdc528932c9 tempest-VolumesSnapshotListTestJSON-2059984684 None] Flow 'volume_create_scheduler' (608e39df-1830-466a-a9a0-50e3f9786f56) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:05:15.715841 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 5, 8, 985592)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:05:15.716446 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:05:15.716623 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:15.717001 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:05:15.720663 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:14.307266 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:05:15.726471 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:05:15.729068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:05:15.730899 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:05:15.732339 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:05:15.734036 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:15.734036 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:15.734036 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:14.307266] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:05:15.734036 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:05:15.734036 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:05:15.737488 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5af9ab57-9fe1-4479-a05a-4af86109e56c tempest-VolumesSnapshotListTestJSON-2059984684 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.24, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:15.732076 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:05:17.049383 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 5, 8, 985592)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:05:17.050039 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:05:17.050237 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:17.050495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:05:17.050872 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.24, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:15.732076 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:05:17.052088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:17.052437 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:17.052798 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.24, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:15.732076] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:05:17.054490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:05:17.054490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:05:17.054867 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cc5df1e1-f118-432e-a5e5-0faae969cbd3 tempest-VolumesSnapshotListTestJSON-2059984684 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:17.054358 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:05:18.536565 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 5, 8, 985592)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:05:18.538854 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:05:18.541347 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:18.544334 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:05:18.544898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:17.054358 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:05:18.550031 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:18.550431 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:18.554611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:17.054358] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:05:18.557740 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:05:18.558213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:05:18.561915 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cb9aac13-d02f-4fdc-9a25-5031a673ddd1 tempest-VolumesSnapshotListTestJSON-2059984684 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.239999999999998, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:18.558266 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:05:27.325226 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1d7f8a6d-7fbe-4299-bedc-512e030252f6 tempest-VolumesSnapshotListTestJSON-2059984684 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:05:54.574889 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cd229302-a1fe-4df3-a6a9-9467066bbec2 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:05:59.265792 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Flow 'volume_create_scheduler' (213b7db7-bd11-47d0-96b0-07ca79ab0311) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:05:59.273806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (dc3a5e8d-e547-44a2-ab14-1f7687a021f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:05:59.276557 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (dc3a5e8d-e547-44a2-ab14-1f7687a021f8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aba81b62-e009-44af-8093-0e9eed95f96c),volume_id=aba81b62-e009-44af-8093-0e9eed95f96c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:05:59.286138 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2e280124-0e2d-4730-9780-e825c600aee2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:05:59.300496 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:05:59.318968 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 5, 27, 318377)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:05:59.319705 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:05:59.320267 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:59.320940 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (aba81b62-e009-44af-8093-0e9eed95f96c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:05:59.321492 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:27.318377 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:05:59.322481 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:05:59.324580 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 344.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:05:59.326649 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 324.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:05:59.328647 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:05:59.332188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:59.332862 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:05:59.333466 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:27.318377] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:05:59.334826 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:05:59.335386 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:05:59.336018 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:05:59.336577 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:05:59.336190 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:05:59.385213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2e280124-0e2d-4730-9780-e825c600aee2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:05:59.393658 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c00fee57-08a4-48c5-9ce4-e923724078eb tempest-AttachVolumeShelveTestJSON-1652031106 None] Flow 'volume_create_scheduler' (213b7db7-bd11-47d0-96b0-07ca79ab0311) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:06:02.687444 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-07cd1eec-1534-4686-b6e4-aaf494c7a53d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:06:54.577401 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-251fb619-b603-4c4e-a2ba-eab9918e0e5c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:07:02.734121 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-22da3f1a-4d03-4caa-867e-b143aff78868 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:07:07.983433 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 2, 727626)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:07.983433 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:07.984277 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:07.984492 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:07.984841 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:07.985303 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:07.985725 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:02.727626] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:07:07.987142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:07.987655 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:07.997829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 2, 727626)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:07.998334 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:07.998963 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:07.999495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:08.000526 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:08.001102 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:08.002193 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:02.727626] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:07:08.002590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:08.002937 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:08.003287 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-7055b83e-03aa-4716-80c7-3b7e6baf3a05 tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:07:09.727441 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (20ef38e9-b89c-4e10-8205-5ea0d54959c4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:09.736293 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8d90a6db-225f-4044-96b0-b68b55b63686) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:09.741519 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8d90a6db-225f-4044-96b0-b68b55b63686) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=38ac0e76-5935-457d-aaab-d080d0d06711,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(be809c79-671b-47f8-ab24-645f1f59aa03),volume_id=be809c79-671b-47f8-ab24-645f1f59aa03,volume_properties=VolumeProperties,volume_type=VolumeType(2a053fcd-7408-4d78-9ffb-8e9cb03d71f3))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:09.750958 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5786dc87-bcfc-4426-b3fc-a3e8194419f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:09.764719 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:07:09.811585 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 2, 727626)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:09.812369 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:09.812940 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:09.814125 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (be809c79-671b-47f8-ab24-645f1f59aa03) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:09.814745 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:02.727626 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:07:09.820764 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:07:09.823105 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:09.825194 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:07:09.829335 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:07:09.835210 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:07:09.835210 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:07:09.838027 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:09.838665 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:09.839266 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:02.727626] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:09.839910 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:09.840351 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:09.841195 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:07:09.841760 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:09.841399 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:09.898703 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5786dc87-bcfc-4426-b3fc-a3e8194419f8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:09.902654 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-74eff8ea-a136-4e07-968a-33177075a19e tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (20ef38e9-b89c-4e10-8205-5ea0d54959c4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:13.876475 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 2, 727626)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:13.876475 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:13.877870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:13.877870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:13.878293 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:09.841399 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:13.880927 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:13.881486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:13.882043 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:09.841399] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:13.882749 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:13.883277 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:13.884158 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-406d5cc2-39dc-4bb7-ba82-ecc2780dd7b9 tempest-GroupSnapshotsTest-2105058445 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:13.883501 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:16.942239 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a6f8c228-a4d5-46de-b43a-6e066f6faed6 tempest-GroupSnapshotsTest-2105058445 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:07:20.853116 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3cbd7085-6805-4741-a320-51edc3540f46 tempest-GroupSnapshotsTest-2105058445 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:07:21.464467 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 20, 848818)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:21.464888 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:21.465364 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:21.465692 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:21.466044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:21.466419 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:21.466802 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:20.848818] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:07:21.467265 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:21.467528 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:21.477403 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 20, 848818)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:21.477818 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:21.478226 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:21.478590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:21.478930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:21.479563 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:21.479704 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:20.848818] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:07:21.480024 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:21.480367 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:21.480711 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-e07d017d-bf12-41cc-b0dd-165d74c1d063 tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:07:23.011367 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (d172c53b-2208-4e0b-a33d-73ccc4fd0bf4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:23.015482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (582ab5b2-3257-4d89-b12d-d346ed78478b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:23.028770 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (582ab5b2-3257-4d89-b12d-d346ed78478b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=1bfbf604-d1ad-4b38-af7a-57db5774c6e4,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a49eba63-c03e-4e1e-b9cf-bf9fdcb2ccdb),volume_id=a49eba63-c03e-4e1e-b9cf-bf9fdcb2ccdb,volume_properties=VolumeProperties,volume_type=VolumeType(adba82d3-33bf-465d-acb0-e70cff3d60bb))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:23.032697 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9ac2c051-6b18-4a1d-8329-c5f39978b625) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:23.045114 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:07:23.066762 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 20, 848818)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:23.067509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:23.068503 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:23.069092 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a49eba63-c03e-4e1e-b9cf-bf9fdcb2ccdb) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:23.069763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:20.848818 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:23.072184 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:23.073843 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 444, in validate_host_capacity Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: backend_state = self.driver.backend_passes_filters( Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:23.075399 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:23.093053 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:23.097986 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:23.097986 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:23.097986 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:20.848818] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:23.097986 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:23.097986 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:23.098897 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:07:23.103338 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:23.089876 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:23.160831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9ac2c051-6b18-4a1d-8329-c5f39978b625) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:23.163445 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ee815ba2-4885-44fc-b7f1-ab7fb626c958 tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (d172c53b-2208-4e0b-a33d-73ccc4fd0bf4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:24.550106 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (0c2d66eb-46e6-4b87-b278-1596f7c99dcd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:24.553206 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c4963fd6-a948-43df-8c20-02cd3f9999c4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:24.555789 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c4963fd6-a948-43df-8c20-02cd3f9999c4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=1bfbf604-d1ad-4b38-af7a-57db5774c6e4,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9ee4f6a9-cd30-4dbc-bf2c-55ef4dfb1169),volume_id=9ee4f6a9-cd30-4dbc-bf2c-55ef4dfb1169,volume_properties=VolumeProperties,volume_type=VolumeType(adba82d3-33bf-465d-acb0-e70cff3d60bb))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:24.558264 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9b928f07-bbc2-4ce7-9f7d-6352f561d917) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:24.565856 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:07:24.605486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 20, 848818)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:24.606078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:24.606623 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:24.606956 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9ee4f6a9-cd30-4dbc-bf2c-55ef4dfb1169) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:24.607357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:23.089876 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:07:24.608411 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:24.610367 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:24.612024 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:07:24.615102 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:07:24.617106 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:24.617106 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:24.617106 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:23.089876] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:24.617106 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:24.617106 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:24.617678 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:07:24.617678 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.24, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:24.610645 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:24.689604 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9b928f07-bbc2-4ce7-9f7d-6352f561d917) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:24.693373 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-aaafad2b-c63a-4ff4-b109-18273ae254f9 tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (0c2d66eb-46e6-4b87-b278-1596f7c99dcd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:37.977239 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a448589-3ea5-4eb9-9299-07fc479435b1 tempest-GroupSnapshotsTest-2105058445 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:07:38.536031 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 37, 972792)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:38.536454 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:38.537149 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:38.537696 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:38.538257 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:38.538610 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:38.539342 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:37.972792] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:07:38.542789 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 384.0, min: 384.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:38.543385 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:38.552134 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 37, 972792)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:38.552668 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:38.553253 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:38.553793 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:38.554346 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:38.554882 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:38.555723 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:37.972792] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:07:38.556348 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 384.0, min: 384.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:38.556915 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:38.557309 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b0a4129b-ab3a-4f1f-bfe5-31c562d96ab2 tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:07:39.906617 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (b12ee147-234e-497c-b373-c6e919ffb09b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:39.910628 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b0156cef-7a29-4da6-a30e-234db6d7beb9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:39.914022 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b0156cef-7a29-4da6-a30e-234db6d7beb9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=5d79fd46-2c9a-4b32-af9c-94cb46735d59,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f2071972-0164-4a37-b7d0-08f99c4d7d65),volume_id=f2071972-0164-4a37-b7d0-08f99c4d7d65,volume_properties=VolumeProperties,volume_type=VolumeType(5d2c235c-bde6-4c08-b2cc-1e735c317062))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:39.916464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8afc63c5-36f0-43ff-934f-0f4cc471f011) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:39.924911 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:07:39.941638 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 37, 972792)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:39.942255 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:39.942629 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:39.942995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f2071972-0164-4a37-b7d0-08f99c4d7d65) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:39.943835 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:37.972792 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:39.944920 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:39.946464 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:07:39.948200 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:39.954895 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:39.958788 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:39.958788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:39.958788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:39.958788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:37.972792] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:39.958788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Weigher CapacityWeigher returned, weigher value is {max: 384.0, min: 384.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:39.959502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:39.959502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:07:39.959502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:39.947178 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:39.982879 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8afc63c5-36f0-43ff-934f-0f4cc471f011) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:39.986736 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-c9f1ad83-2be1-48bc-a912-d8ea6c6285a5 tempest-GroupSnapshotsTest-2105058445 None] Flow 'volume_create_scheduler' (b12ee147-234e-497c-b373-c6e919ffb09b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:44.649117 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-997013bf-24b6-4aff-b817-4132f7f741d1 tempest-GroupSnapshotsTest-2105058445 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:07:48.897669 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 44, 642539)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:48.898407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:48.898586 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:48.898948 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:48.899843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:48.900210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:48.900587 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:44.642539] {{(pid=133499) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Nov 20 21:07:48.901531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Weigher CapacityWeigher returned, weigher value is {max: 383.0, min: 383.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:48.902443 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:48.913376 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 44, 642539)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:48.913797 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:48.914798 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:48.915281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:48.915635 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:48.915992 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:48.916808 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:44.642539] {{(pid=133499) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Nov 20 21:07:48.917226 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Weigher CapacityWeigher returned, weigher value is {max: 383.0, min: 383.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:48.917708 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:48.918209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a1f895ff-b795-41fa-8eb6-19ff2831c816 tempest-GroupsV314Test-1131188880 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Nov 20 21:07:50.268006 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Flow 'volume_create_scheduler' (19c46523-90df-41cd-83a0-1bf7ca8c1a52) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:50.280870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (90ab64db-58e2-4068-8fd3-af8fe8fc6943) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:50.284213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (90ab64db-58e2-4068-8fd3-af8fe8fc6943) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=61dfce34-c9a4-45ed-a457-372182b91f26,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(768b045e-52e5-4222-9fb0-063efb032b16),volume_id=768b045e-52e5-4222-9fb0-063efb032b16,volume_properties=VolumeProperties,volume_type=VolumeType(24fde958-dc49-4efa-9ad6-327039156311))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:50.285980 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8f7fe2ad-5821-4bee-98d5-0244243578ad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:07:50.292833 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:07:50.309285 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 44, 642539)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:50.309703 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:50.310084 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:50.310460 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (768b045e-52e5-4222-9fb0-063efb032b16) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:50.310857 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:44.642539 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:50.312244 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:50.312616 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:50.313008 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:44.642539] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:50.313407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Weigher CapacityWeigher returned, weigher value is {max: 383.0, min: 383.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:50.313752 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:50.314135 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:07:50.314510 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:50.314160 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:50.344089 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8f7fe2ad-5821-4bee-98d5-0244243578ad) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:07:50.346883 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ec958d65-b82f-4b28-8b90-29dc00c620bc tempest-GroupsV314Test-1131188880 None] Flow 'volume_create_scheduler' (19c46523-90df-41cd-83a0-1bf7ca8c1a52) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:07:51.683751 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 7, 44, 642539)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:07:51.684469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:07:51.684469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:51.684825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:07:51.685423 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:50.314160 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:51.686267 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:07:51.688488 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:07:51.690839 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:07:51.692489 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:07:51.694554 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:07:51.695722 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:51.696091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:07:51.696464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:50.314160] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:07:51.696873 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Weigher CapacityWeigher returned, weigher value is {max: 382.0, min: 382.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:07:51.697216 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:07:51.697861 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a750223-9944-44b1-8b34-8fdaa481d964 tempest-GroupsV314Test-1131188880 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.24, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:07:51.697436 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:07:54.575827 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7aa15ff9-5524-4111-a0c4-babffc8488be None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:07:54.843626 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-de9fa550-e4b5-40b8-92ec-0af0a313d528 tempest-GroupsV314Test-1131188880 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:07:55.605682 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-09c439bc-9514-486e-81f7-cec500784f54 tempest-GroupsV314Test-1131188880 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:08:02.812804 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8cd25550-db8e-436d-827a-b9d2647a33ca None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:08:07.311318 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Flow 'volume_create_scheduler' (1d00af11-f0b3-43a4-bf8c-d6540f62a4b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:07.313729 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2bff563f-82ac-4042-9375-5ca42812f99b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:07.316412 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2bff563f-82ac-4042-9375-5ca42812f99b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(76714f42-c4f7-48b6-a322-9705e5401164),volume_id=76714f42-c4f7-48b6-a322-9705e5401164,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:07.318572 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4df24e7c-21d6-4e33-8192-abca964eb9e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:07.326659 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:08:07.343060 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 8, 2, 808657)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:08:07.343664 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:08:07.344416 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:07.345419 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (76714f42-c4f7-48b6-a322-9705e5401164) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:08:07.346742 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:02.808657 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 444, in validate_host_capacity Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: backend_state = self.driver.backend_passes_filters( Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:08:07.358814 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:08:07.360588 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:08:07.362853 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:08:07.364338 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:08:07.370181 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:07.370590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:07.371165 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:02.808657] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:08:07.371801 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:08:07.372153 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:08:07.372648 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:08:07.373027 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.24, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:07.372678 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:08:07.402294 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4df24e7c-21d6-4e33-8192-abca964eb9e1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:07.404426 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ad6b9001-a1d7-4483-9c10-60a592e0ad5f tempest-VolumeQuotasAdminTestJSON-1057255691 None] Flow 'volume_create_scheduler' (1d00af11-f0b3-43a4-bf8c-d6540f62a4b0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:09.237681 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fe8b5b57-2970-4fc7-8a9c-dbe27336aeb8 tempest-VolumeQuotasAdminTestJSON-1057255691 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:08:09.962391 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Flow 'volume_create_scheduler' (169c5be4-f5ff-4052-8204-834befc9aab2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:09.964848 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (08aea6ad-0d9b-4d03-9338-8e74826a102f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:09.967332 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (08aea6ad-0d9b-4d03-9338-8e74826a102f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(40b66e6d-b877-4a13-a3f0-7b0f8b2d531c),volume_id=40b66e6d-b877-4a13-a3f0-7b0f8b2d531c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:09.969421 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f80dc372-8e80-4271-a595-45e5bde67f8a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:09.977223 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:08:09.993859 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 8, 9, 232845)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:08:09.994572 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:08:09.995084 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:09.995605 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (40b66e6d-b877-4a13-a3f0-7b0f8b2d531c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:08:09.996316 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:09.232845 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:08:09.998091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:09.998666 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:09.999214 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:09.232845] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:08:09.999827 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Weigher CapacityWeigher returned, weigher value is {max: 380.79999999999995, min: 380.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:08:10.000372 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:08:10.001021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:08:10.001712 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:10.001223 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:08:10.032698 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f80dc372-8e80-4271-a595-45e5bde67f8a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:10.035342 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-4d0e057d-acc5-4be5-b70a-0ebe82036c9b tempest-VolumeQuotasAdminTestJSON-1057255691 None] Flow 'volume_create_scheduler' (169c5be4-f5ff-4052-8204-834befc9aab2) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:12.681773 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c4b0a249-19cc-4597-b0f0-0cc93c65d0ae tempest-VolumeQuotasAdminTestJSON-2794273 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:08:28.689323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Flow 'volume_create_scheduler' (ca02d16e-0db8-4600-8523-114c7e60c2d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:28.691984 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b7c0aa97-523b-49f3-af70-41b089e8990b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:28.708899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b7c0aa97-523b-49f3-af70-41b089e8990b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(11bef6fc-4825-4a25-ad07-79da6da17519),volume_id=11bef6fc-4825-4a25-ad07-79da6da17519,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:28.711250 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (76a09031-3c31-4d07-abb1-8921679b3f8a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:28.730637 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:08:28.759188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 8, 12, 677158)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:08:28.760494 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:08:28.761601 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:28.762105 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (11bef6fc-4825-4a25-ad07-79da6da17519) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:08:28.762547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:12.677158 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:08:28.763661 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:08:28.765571 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:08:28.768697 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:08:28.770778 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:08:28.774185 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:08:28.774185 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:08:28.774185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:28.774185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:28.774185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:12.677158] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:08:28.774185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:08:28.775045 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:08:28.775045 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:08:28.775045 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:28.766091 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:08:28.808139 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (76a09031-3c31-4d07-abb1-8921679b3f8a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:28.810187 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-225a2012-67a9-45bc-b82c-f5d4d898e923 tempest-VolumesTransfersTest-1100044084 None] Flow 'volume_create_scheduler' (ca02d16e-0db8-4600-8523-114c7e60c2d2) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:31.371502 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-01c04bfe-fcab-4104-8b64-cca03c8b5279 tempest-VolumesTransfersTest-547973625 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:08:32.162546 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Flow 'volume_create_scheduler' (a5e89435-5471-424d-9a86-563ee0c0b38a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:32.165868 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0f8aaf7a-43a6-4d45-833d-dc21fe71532b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:32.169158 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0f8aaf7a-43a6-4d45-833d-dc21fe71532b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9dcb53fa-22c8-4bda-a1ec-a6969f19fa71),volume_id=9dcb53fa-22c8-4bda-a1ec-a6969f19fa71,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:32.171344 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2ee86380-814f-4b1d-b1f4-ed73fbf35770) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:08:32.179798 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:08:32.198589 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 8, 31, 364357)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:08:32.199246 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:08:32.199615 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:32.199994 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9dcb53fa-22c8-4bda-a1ec-a6969f19fa71) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:08:32.200373 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:31.364357 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:08:32.201410 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:08:32.202979 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:08:32.204594 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:08:32.207539 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:08:32.212310 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:08:32.212310 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:32.212310 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:08:32.212310 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:31.364357] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:08:32.212310 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:08:32.213427 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:08:32.213427 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:08:32.213427 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:08:32.203835 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:08:32.242164 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2ee86380-814f-4b1d-b1f4-ed73fbf35770) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:08:32.244111 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-906ff245-9e5a-4042-a09b-3caed9ebd4f8 tempest-VolumesTransfersTest-1100044084 None] Flow 'volume_create_scheduler' (a5e89435-5471-424d-9a86-563ee0c0b38a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:08:34.081639 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9921adf2-51b0-47aa-a618-052440d531cd tempest-VolumesTransfersTest-547973625 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:08:54.569231 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e3948a29-3715-49be-873c-b483ad0b5f71 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:09:02.741197 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a88e6461-6ece-4000-b8cd-dc4ae9741411 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:09:53.651843 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Flow 'volume_create_scheduler' (ad8fae0f-3d7b-4038-825c-a3f301c8202b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:09:53.654635 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a6519812-6013-4598-b7ee-618ad5e8f417) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:09:53.657634 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a6519812-6013-4598-b7ee-618ad5e8f417) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4972b21b-fa11-4ec9-8890-516259485f08),volume_id=4972b21b-fa11-4ec9-8890-516259485f08,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:09:53.659606 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b1d3bc2f-bd4a-47bf-86d4-69481f5e1daa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:09:53.666949 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:09:53.690697 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 9, 2, 736368)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:09:53.691109 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:09:53.691469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:53.691824 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4972b21b-fa11-4ec9-8890-516259485f08) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:09:53.692214 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:02.736368 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:09:53.693287 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:09:53.695098 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:09:53.697519 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:09:53.699898 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:09:53.701965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:53.701965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:53.701965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:02.736368] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:09:53.701965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:09:53.701965 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:09:53.702451 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:09:53.702451 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:53.695571 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:09:53.723995 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b1d3bc2f-bd4a-47bf-86d4-69481f5e1daa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:09:53.726131 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-2ce1893c-5715-4b53-b8ef-7f8aac30aa87 tempest-AttachVolumeShelveTestJSON-1652031106 None] Flow 'volume_create_scheduler' (ad8fae0f-3d7b-4038-825c-a3f301c8202b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:09:54.574423 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5516a367-dbe7-48e9-bc33-cef169797082 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:09:58.321002 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Flow 'volume_create_scheduler' (e637e8fe-1682-46e2-a025-6eccd429f5da) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:09:58.323492 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (084a2355-31e7-4fea-9113-5858d3328af8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:09:58.325911 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (084a2355-31e7-4fea-9113-5858d3328af8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ee2bda93-8a88-4348-b5c4-b7be1aa0cea9),volume_id=ee2bda93-8a88-4348-b5c4-b7be1aa0cea9,volume_properties=VolumeProperties,volume_type=VolumeType(fa9b161a-9b4b-4340-9ef4-e8f0fa410783))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:09:58.328573 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ab600796-301c-4f26-8379-8f2693e1473b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:09:58.336103 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:09:58.351648 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 9, 2, 736368)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:09:58.352040 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:09:58.352402 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:58.352760 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ee2bda93-8a88-4348-b5c4-b7be1aa0cea9) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:09:58.353155 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:53.695571 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:09:58.354555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:58.354945 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:58.355313 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:53.695571] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:09:58.355727 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Weigher CapacityWeigher returned, weigher value is {max: 380.0, min: 380.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:09:58.356070 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:09:58.356533 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:09:58.356904 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:58.356560 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:09:58.385504 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ab600796-301c-4f26-8379-8f2693e1473b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:09:58.387571 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-545b54d1-e184-46a9-abb8-97f0f673fad1 tempest-VolumeTypesTest-1448565603 None] Flow 'volume_create_scheduler' (e637e8fe-1682-46e2-a025-6eccd429f5da) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:09:59.579168 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 9, 2, 736368)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:09:59.579827 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:09:59.579935 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:59.580257 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:59.580636 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:09:59.581016 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:09:58.356560] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:09:59.581418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Weigher CapacityWeigher returned, weigher value is {max: 379.0, min: 379.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:09:59.581880 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:09:59.723518 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1e892e3a-27c6-48ad-805b-c4513c03cce2 tempest-VolumeTypesTest-1448565603 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:01.854309 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a32d1fdd-1da9-49c6-8e26-83ccf5c39d2d tempest-VolumeTypesTest-1448565603 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:02.648963 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f19d9978-14c5-4d4d-a30b-c0e17abbdee2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:04.899655 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Flow 'volume_create_scheduler' (f0d65492-d04e-4145-b57f-da159785a970) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:04.902194 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f73a4072-005b-4b5c-96ea-ce1b124f8512) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:04.904647 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f73a4072-005b-4b5c-96ea-ce1b124f8512) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(df8f9429-cdd5-4d82-bc7a-4646e37c431b),volume_id=df8f9429-cdd5-4d82-bc7a-4646e37c431b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:04.906698 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (07ff4b63-a10d-42a3-98fc-3e1ff8084e29) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:04.914876 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:10:04.934142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 10, 2, 645019)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:10:04.934142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:10:04.934142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:04.934142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (df8f9429-cdd5-4d82-bc7a-4646e37c431b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:02.645019 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:10:04.934541 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:10:04.936886 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:10:04.938878 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:04.940560 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:04.942830 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:02.645019] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:10:04.944420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:10:04.944420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:10:04.944420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:10:04.944420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:04.935435 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:10:04.962722 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (07ff4b63-a10d-42a3-98fc-3e1ff8084e29) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:04.964744 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-5fb1f16c-4f8f-4f8c-93bb-489aed5e8d1a tempest-VolumesMetadataTest-446331414 None] Flow 'volume_create_scheduler' (f0d65492-d04e-4145-b57f-da159785a970) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:06.933978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d45a68e5-4532-4010-8551-ffe791e4ceea tempest-VolumesMetadataTest-446331414 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:09.440649 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Flow 'volume_create_scheduler' (5431a95b-8e53-4524-a259-0b70566b793c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:09.443275 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cc063de9-7188-4e2e-85c7-23c16911d1f3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:09.445962 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cc063de9-7188-4e2e-85c7-23c16911d1f3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7f30b312-663e-4f3c-adca-e7ac0a60a8df),volume_id=7f30b312-663e-4f3c-adca-e7ac0a60a8df,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:09.448015 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fc457565-5fcd-4217-b1b6-55a55e4c8caa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:09.456475 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:10:09.471913 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 10, 6, 929716)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:10:09.471913 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:10:09.472149 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:09.472379 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (7f30b312-663e-4f3c-adca-e7ac0a60a8df) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:10:09.472771 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:06.929716 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:10:09.473816 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:10:09.475572 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:10:09.478025 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:10:09.483972 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:10:09.488518 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:10:09.488518 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:09.488518 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:09.488518 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:06.929716] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:10:09.488518 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:10:09.489359 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:10:09.489359 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:10:09.489359 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:09.478926 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:10:09.508897 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fc457565-5fcd-4217-b1b6-55a55e4c8caa) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:09.511438 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-31599289-9132-497e-ac1c-168aa7d53496 tempest-VolumesCloneNegativeTest-861478784 None] Flow 'volume_create_scheduler' (5431a95b-8e53-4524-a259-0b70566b793c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:11.217064 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dc09ace8-9c60-4eb0-b96d-6c76150b5312 tempest-VolumesCloneNegativeTest-861478784 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:15.806596 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Flow 'volume_create_scheduler' (0f351666-8f84-4ad7-a8ad-dc8c8bba9411) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:15.809052 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (08efd8cf-9358-419a-abd1-1b1c42193220) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:15.812296 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (08efd8cf-9358-419a-abd1-1b1c42193220) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ed534213-bcdf-4f81-8863-6437d9b5bf41),volume_id=ed534213-bcdf-4f81-8863-6437d9b5bf41,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:15.814513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (627204df-fc6f-49b7-81ce-73ce11b526f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:15.824996 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:10:15.839452 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 10, 11, 209940)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:10:15.839869 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:10:15.840230 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:15.840585 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ed534213-bcdf-4f81-8863-6437d9b5bf41) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:10:15.840976 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:11.209940 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:10:15.842635 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:10:15.844625 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:10:15.846445 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:10:15.848110 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:10:15.849673 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:15.849673 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:15.849673 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:11.209940] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:10:15.849673 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:10:15.849673 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:10:15.851152 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:10:15.851152 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:15.846706 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:10:15.880898 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (627204df-fc6f-49b7-81ce-73ce11b526f5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:15.883497 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-d1a293f8-9db9-4252-a261-e60debbc56dc tempest-VolumesNegativeTest-452759441 None] Flow 'volume_create_scheduler' (0f351666-8f84-4ad7-a8ad-dc8c8bba9411) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:18.087282 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Flow 'volume_create_scheduler' (d0ebaa51-88ef-4689-8691-fcb4c14350e7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:18.092489 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (68f59a73-71fc-4f16-9d7b-37c422976f28) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:18.095723 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (68f59a73-71fc-4f16-9d7b-37c422976f28) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(afabfac2-5807-4d77-9ff3-9126704d7471),volume_id=afabfac2-5807-4d77-9ff3-9126704d7471,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:18.098171 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (312b60b2-2869-408a-ad58-fa3a7e7bd8b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:18.110690 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:10:18.128532 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 10, 11, 209940)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:10:18.129201 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:10:18.129873 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:18.130467 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (afabfac2-5807-4d77-9ff3-9126704d7471) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:10:18.131074 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:15.846706 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:10:18.133025 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:18.133775 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:18.134387 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:15.846706] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:10:18.135004 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:10:18.135555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:10:18.136212 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:10:18.136782 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:18.136426 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:10:18.174611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (312b60b2-2869-408a-ad58-fa3a7e7bd8b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:18.177008 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-ed075faf-4cde-41d9-af95-2ae58ef800c1 tempest-AttachVolumeTestJSON-1552858705 None] Flow 'volume_create_scheduler' (d0ebaa51-88ef-4689-8691-fcb4c14350e7) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:34.656189 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-26408ba8-2fc6-494e-9f29-7e7d3f866143 tempest-VolumesNegativeTest-452759441 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:38.120257 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (7e372578-e6ed-4c07-af68-78ef4cc234ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:38.123460 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (25c21812-cfe3-4d92-9263-ee73029c1db3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:38.125959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (25c21812-cfe3-4d92-9263-ee73029c1db3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9caac17a-9de9-446c-8f80-5ea71b8d2ccc),volume_id=9caac17a-9de9-446c-8f80-5ea71b8d2ccc,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:38.127925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (384eeb7a-4909-44b9-b64e-18f9cd6e17df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:10:38.135145 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:10:38.153486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 10, 34, 651483)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:10:38.153486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:10:38.153486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:38.153486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9caac17a-9de9-446c-8f80-5ea71b8d2ccc) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:34.651483 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:10:38.153941 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:10:38.156646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:10:38.158747 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:10:38.160549 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:38.164983 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:34.651483] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:10:38.166480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:10:38.166480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:10:38.166480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:10:38.166480 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:38.154820 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:10:38.183961 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (384eeb7a-4909-44b9-b64e-18f9cd6e17df) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:10:38.186670 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-022291be-5d12-453a-92f1-124239d1025c tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (7e372578-e6ed-4c07-af68-78ef4cc234ee) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:10:39.448458 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 10, 34, 651483)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:10:39.450182 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:10:39.450730 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:39.451403 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:10:39.452077 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:38.154820 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:10:39.454837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:39.455408 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:10:39.456117 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:38.154820] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:10:39.456912 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 377.79999999999995, min: 377.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:10:39.458530 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:10:39.459142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d8a34af5-21ee-4816-af6b-1ecff5a31723 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.79,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:10:39.458785 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:10:41.734341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-7724766f-b659-460e-8608-d3a4473c1040 tempest-VolumesSnapshotTestJSON-1178343811 None] create_backup in rpcapi backup_id 98b2fc72-ce3f-4a30-a005-b63af66fd059 {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:10:48.850858 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d4ba320b-aa19-47e2-9784-5fdea1971f8a tempest-TaggedBootDevicesTest-1735451821 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:49.938951 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ddf91758-8b8d-4634-bd45-cb36bb64c3b2 tempest-TaggedBootDevicesTest-1735451821 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.66, 'free_capacity_gb': 19.1, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:10:51.860521 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6e4c3acd-29db-464a-8c91-f50d58b7adf1 tempest-TaggedBootDevicesTest-1735451821 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.66, 'free_capacity_gb': 19.1, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:11:02.680994 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-13ea39ef-4d16-48f7-a59c-f9c8dbd28dcd None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:11:10.948974 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (012e56fd-0b36-4770-abcd-e7743ee44807) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:10.951493 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1fdb509a-813b-4ed6-af45-6bfcd636bce2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:10.953952 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1fdb509a-813b-4ed6-af45-6bfcd636bce2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8cf4195c-9738-4402-9871-a54b9aa3916c),volume_id=8cf4195c-9738-4402-9871-a54b9aa3916c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:10.955939 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (232563fc-25e0-441d-9f37-26c11114203a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:10.963427 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:11:10.982490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 2, 668656)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:10.982490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:10.982490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:10.982490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (8cf4195c-9738-4402-9871-a54b9aa3916c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:02.668656 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:11:10.982968 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:10.984370 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:10.986435 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:10.988736 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:10.990618 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:10.991851 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:02.668656] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:10.991851 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:10.991851 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:10.991851 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:11:10.992202 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:10.983886 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:11.009792 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (232563fc-25e0-441d-9f37-26c11114203a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:11.011901 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-55132864-bff1-4bba-a275-cd62d8e28c58 tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (012e56fd-0b36-4770-abcd-e7743ee44807) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:14.613515 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 2, 668656)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:14.614112 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:14.614499 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:14.614607 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:14.614940 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:10.983886 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:11:14.615742 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:11:14.617855 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:14.619617 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:14.621924 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:14.625021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:14.625021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:14.625021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:10.983886] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:14.625021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 380.0, min: 380.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:14.625021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:14.625733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-250d5814-9e37-4bad-a793-40378887a232 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:14.617518 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:16.856857 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 2, 668656)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:16.858694 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:16.859287 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:16.859655 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:16.860038 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:14.617518 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:16.861296 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:16.861528 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:16.861897 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:14.617518] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:16.862506 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 379.0, min: 379.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:16.862852 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:16.863247 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5e101363-9efc-4900-b15c-cbbbabf27b6d tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.8,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:16.862898 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:18.020123 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 2, 668656)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:18.020718 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:18.020859 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:18.021210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:18.021577 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.8,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:16.862898 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:18.022732 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:18.023080 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:18.023439 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.8,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:16.862898] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:18.023974 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 378.0, min: 378.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:18.024314 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:18.024710 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a2104995-662d-45b6-be8e-4d3d5944776e tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.8,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:18.024355 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:25.472594 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bb24f96b-9b36-4ed1-97a2-11dbaf7ec84a tempest-AttachVolumeShelveTestJSON-1652031106 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:11:26.735474 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7d8807b2-8363-4465-a5c4-e0b5a946b463 tempest-AttachVolumeShelveTestJSON-1652031106 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:11:46.626542 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Flow 'volume_create_scheduler' (e99ecbf8-9bb0-4c02-8160-7ebeb0542f26) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:46.629083 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f7365998-96c5-4064-96d5-d26f5a36ff0d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:46.631893 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f7365998-96c5-4064-96d5-d26f5a36ff0d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(6378e118-02e2-45ce-8e30-a5233665db32),volume_id=6378e118-02e2-45ce-8e30-a5233665db32,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:46.633870 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a92ccdd1-7062-4069-a91d-07bb6483f884) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:46.641688 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:11:46.656720 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 26, 730364)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:46.657227 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:46.657593 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:46.657957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (6378e118-02e2-45ce-8e30-a5233665db32) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:46.658381 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.8,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:26.730364 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:46.659438 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:46.661938 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:46.664125 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 326.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:46.666385 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:46.668677 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:46.668677 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:46.668677 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:46.668677 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:46.668677 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.8,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:26.730364] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:46.668677 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Weigher CapacityWeigher returned, weigher value is {max: 382.0, min: 382.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:46.669172 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:46.669172 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:11:46.669172 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:46.661918 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:46.687394 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a92ccdd1-7062-4069-a91d-07bb6483f884) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:46.689415 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8ab427db-7cd3-47d1-be46-d74ba6f44cdf tempest-AttachVolumeNegativeTest-1315300571 None] Flow 'volume_create_scheduler' (e99ecbf8-9bb0-4c02-8160-7ebeb0542f26) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:47.827132 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Flow 'volume_create_scheduler' (2d086412-e9c0-45ab-99f4-4f1ae8f2cbea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:47.829663 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4e1db302-fa69-47c5-b084-d07e33ef5e99) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:47.832151 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4e1db302-fa69-47c5-b084-d07e33ef5e99) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(03759c8b-d076-4325-a9e8-74ab1406e8a5),volume_id=03759c8b-d076-4325-a9e8-74ab1406e8a5,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:47.834316 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c1168eb4-e859-498b-85b5-5a19b81ba84f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:47.842801 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:11:47.857531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 26, 730364)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:47.857927 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:47.858323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:47.858818 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (03759c8b-d076-4325-a9e8-74ab1406e8a5) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:47.859194 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:46.661918 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:11:47.860532 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:47.862745 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:47.865119 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:11:47.867935 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:11:47.870856 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:11:47.870856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:47.870856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:47.870856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:46.661918] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:47.870856 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:47.871281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:47.871281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:11:47.871281 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:47.862992 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:47.897781 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c1168eb4-e859-498b-85b5-5a19b81ba84f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:47.899831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-02093241-4dee-4d6a-8a1c-55ba5b570845 tempest-ServersNegativeTestJSON-1010955703 None] Flow 'volume_create_scheduler' (2d086412-e9c0-45ab-99f4-4f1ae8f2cbea) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:49.679797 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Flow 'volume_create_scheduler' (f028b9e9-fbee-4225-9498-946501b169d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:49.682486 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (365e2c27-235e-4507-9014-41937fddec3d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:49.685479 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (365e2c27-235e-4507-9014-41937fddec3d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(508f276a-ae0f-4414-bedc-655e8fe45fb4),volume_id=508f276a-ae0f-4414-bedc-655e8fe45fb4,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:49.687862 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ec3057f7-b7b2-450b-a60f-01165f344c61) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:11:49.695236 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:11:49.709207 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 11, 26, 730364)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:11:49.709615 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:11:49.709978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:49.710364 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (508f276a-ae0f-4414-bedc-655e8fe45fb4) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:11:49.710763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:47.862992 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:49.711803 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:11:49.713731 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:11:49.716216 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:11:49.718213 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:11:49.720297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:49.720297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:11:49.720297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:47.862992] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:11:49.720297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Weigher CapacityWeigher returned, weigher value is {max: 380.0, min: 380.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:11:49.720297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:11:49.720737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:11:49.720737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:11:49.714229 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:11:49.740349 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ec3057f7-b7b2-450b-a60f-01165f344c61) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:11:49.742420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8950d707-6dca-4139-b497-012f77b652a7 tempest-ServersNegativeTestJSON-1010955703 None] Flow 'volume_create_scheduler' (f028b9e9-fbee-4225-9498-946501b169d8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:11:54.584567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fdefe5c5-e2ed-4553-a6b3-8dcb39a07910 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:12:02.664096 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-60a41052-d9e1-4421-99d2-b3f7105e8df2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:12:14.715999 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Flow 'volume_create_scheduler' (3ca6b6bf-4b71-4618-9aa2-ccfcf30df978) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:12:14.719157 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f4596f14-95f8-475d-bf4a-2012a6ba9a48) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:12:14.721619 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f4596f14-95f8-475d-bf4a-2012a6ba9a48) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4cca5e46-96e2-4959-bc27-d34e2329c021),volume_id=4cca5e46-96e2-4959-bc27-d34e2329c021,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:12:14.723576 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (67a0eab8-5c82-4eca-9349-1f981d722c59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:12:14.731533 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:12:14.749606 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 12, 2, 654379)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:12:14.750407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:12:14.751002 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:14.751574 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4cca5e46-96e2-4959-bc27-d34e2329c021) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:12:14.752459 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:02.654379 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:12:14.754270 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:14.755509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:14.756308 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.8,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:02.654379] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:12:14.756997 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Weigher CapacityWeigher returned, weigher value is {max: 379.0, min: 379.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:12:14.757769 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:12:14.758642 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:12:14.759412 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:14.758907 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:12:14.786712 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (67a0eab8-5c82-4eca-9349-1f981d722c59) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:12:14.788890 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a5026d6-7126-4e10-85f7-aa508ac23c83 tempest-AttachVolumeNegativeTest-1315300571 None] Flow 'volume_create_scheduler' (3ca6b6bf-4b71-4618-9aa2-ccfcf30df978) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:12:31.979088 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Flow 'volume_create_scheduler' (52bd1cbf-2b95-47d1-9b7f-e2faf45e59cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:12:31.982676 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9a7c0b37-e92b-4dc9-b387-2cef21415dc9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:12:31.986500 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9a7c0b37-e92b-4dc9-b387-2cef21415dc9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9a35edc4-f036-418e-9ab7-1dc17c833f8c),volume_id=9a35edc4-f036-418e-9ab7-1dc17c833f8c,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:12:31.989333 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a1aafc0c-a5e2-470b-b615-f6ffe08540a1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:12:31.997381 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:12:32.022355 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.8, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 12, 2, 654379)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:12:32.022750 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:12:32.023114 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:32.023474 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9a35edc4-f036-418e-9ab7-1dc17c833f8c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:12:32.023877 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:14.758907 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:12:32.025304 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:12:32.027385 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:12:32.029005 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:12:32.031195 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:12:32.033478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:32.033478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:32.033478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.8,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:14.758907] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:12:32.033478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Weigher CapacityWeigher returned, weigher value is {max: 378.0, min: 378.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:12:32.033478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:12:32.033925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:12:32.033925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.8,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:32.028109 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:12:32.061142 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a1aafc0c-a5e2-470b-b615-f6ffe08540a1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:12:32.063564 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-82ca601e-62a0-48bf-a00a-ec526edb6303 tempest-AttachVolumeNegativeTest-1315300571 None] Flow 'volume_create_scheduler' (52bd1cbf-2b95-47d1-9b7f-e2faf45e59cd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:12:37.671713 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-758b0eb6-cd6e-4084-921d-c538ee9b1577 tempest-AttachVolumeNegativeTest-1315300571 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 18, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:12:42.538028 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-85e4ff1c-04ad-4da2-8cf5-06a9cb56bec1 tempest-AttachVolumeNegativeTest-1315300571 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:12:54.591337 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-17441288-c3f0-4b93-8c33-b4377309c957 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:12:57.017990 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0370f7d1-9895-4d36-9575-745cabc77fdb tempest-AttachVolumeNegativeTest-1315300571 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:12:57.100035 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 12, 57, 12491)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:12:57.100371 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:12:57.100739 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:57.101120 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:12:57.101495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:57.012491 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:12:57.102686 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:57.103038 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:12:57.103611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:57.012491] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:12:57.104022 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:12:57.104375 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:12:57.104770 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-94e8a086-bf1d-4aa7-9bf5-cd9c4d19649b tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:57.104421 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:13:00.506247 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 12, 57, 12491)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:13:00.506913 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:13:00.507005 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:00.507751 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:13:00.508206 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:57.104421 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:13:00.508739 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:13:00.510491 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:13:00.513350 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:13:00.517926 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:13:00.522325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:00.522325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:00.522325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:12:57.104421] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:13:00.522325 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:13:00.522861 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:13:00.522861 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5b5c5bb8-a510-4010-90cd-aa1a92972118 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:00.511095 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:13:02.671916 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c45d7e64-404d-4428-991a-ada43f558050 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:13:08.290263 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-86024d3c-acb6-4173-8ad3-b796f81566a6 tempest-ServersNegativeTestJSON-1010955703 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:13:09.521364 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-aadb4104-bff1-4358-8768-f5a53bd944f5 tempest-ServersNegativeTestJSON-1010955703 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:13:15.720813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 9, 513844)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:13:15.722242 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:13:15.722838 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:15.723588 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:13:15.724368 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:09.513844 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:13:15.725804 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:15.726463 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:15.726930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:09.513844] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:13:15.727820 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 381.79999999999995, min: 381.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:13:15.728393 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:13:15.729006 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6502d482-8777-48b8-8f35-7d8509449122 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:15.728653 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:13:16.948788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 9, 513844)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:13:16.949976 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:13:16.950523 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:16.951044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:13:16.951583 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:15.728653 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:13:16.952879 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:13:16.955468 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:13:16.958823 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:13:16.961264 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:13:16.969820 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:13:16.969820 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:13:16.969820 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:16.970957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:16.973356 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:15.728653] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:13:16.974238 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 380.79999999999995, min: 380.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:13:16.974806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:13:16.975391 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-64ac958a-c261-4ce9-82d9-c609a6893076 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:16.975031 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:13:36.333194 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Flow 'volume_create_scheduler' (f6b74805-ab79-4bcb-9419-9df62b14bd64) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:13:36.336008 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b380c7b7-ee97-4ebe-8ccf-58b19843e2b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:13:36.338626 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b380c7b7-ee97-4ebe-8ccf-58b19843e2b0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(100fba8c-d0ca-483f-adbb-d4f0bae26a32),volume_id=100fba8c-d0ca-483f-adbb-d4f0bae26a32,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:13:36.340837 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4454175c-ed7f-40fc-9e76-36e814bfbf2e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:13:36.350642 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:13:36.366277 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 9, 513844)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:13:36.366722 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:13:36.367311 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:36.367675 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (100fba8c-d0ca-483f-adbb-d4f0bae26a32) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:13:36.368078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:16.975031 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:13:36.370155 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:36.370555 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:36.370932 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:16.975031] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:13:36.371534 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:13:36.371885 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:13:36.372354 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:13:36.372737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:36.372384 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:13:36.405451 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4454175c-ed7f-40fc-9e76-36e814bfbf2e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:13:36.408549 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-10f0a852-d37f-498a-bf31-81bcbc20133e req-f0bebd49-4eb8-4015-adef-e7c5939347cd tempest-VolumesSnapshotsTestJSON-1979916101 None] Flow 'volume_create_scheduler' (f6b74805-ab79-4bcb-9419-9df62b14bd64) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:13:37.689737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 9, 513844)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:13:37.690942 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:13:37.691311 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:37.691675 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:13:37.692053 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:36.372384 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:13:37.693054 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:13:37.695449 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:13:37.698575 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:13:37.701035 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:13:37.705514 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:13:37.705514 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:13:37.705514 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:37.706370 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:13:37.706789 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:36.372384] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:13:37.707210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:13:37.707566 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:13:37.707967 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-e1a67f39-ee5c-4937-9516-f2d981c29b76 req-bbe54b41-85e8-43f5-aaa4-f3fd7808a3b0 tempest-VolumesSnapshotsTestJSON-1979916101 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:37.707618 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:13:40.499667 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-075f2c06-fc03-4c45-827d-43674aa1d383 req-85535b5e-9f13-448f-a93b-f3eb321bdb04 tempest-VolumesSnapshotsTestJSON-1979916101 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:13:54.598999 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b970351c-b551-4433-bd1d-73a825b87679 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:14:06.151132 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Flow 'volume_create_scheduler' (e67484e3-af2e-4a63-b12c-63e1421cc839) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:06.153634 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (884af962-4fc0-4e06-888a-565c31e3a0d2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:06.160070 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (884af962-4fc0-4e06-888a-565c31e3a0d2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ff002351-c5f6-452e-8aa5-9a714abe58eb),volume_id=ff002351-c5f6-452e-8aa5-9a714abe58eb,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:06.162346 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (52c8a3ce-642f-4535-9711-d4059075c194) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:06.171100 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:06.196289 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:06.196289 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:06.196289 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:06.196289 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ff002351-c5f6-452e-8aa5-9a714abe58eb) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:40.494914 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:06.196742 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 326.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:06.198252 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:14:06.200674 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:06.204191 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:06.209611 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:13:40.494914] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:06.210959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Weigher CapacityWeigher returned, weigher value is {max: 381.79999999999995, min: 381.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:06.210959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:06.210959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:06.210959 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:06.193384 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:06.224221 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (52c8a3ce-642f-4535-9711-d4059075c194) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:06.226379 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-79627d60-74b6-40e1-847d-8993973ff4af tempest-AttachVolumeTestJSON-1552858705 None] Flow 'volume_create_scheduler' (e67484e3-af2e-4a63-b12c-63e1421cc839) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:07.937923 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-eb0cf1d2-bcd6-41fd-a296-2199f4d94d0d tempest-BackendsCapabilitiesAdminTestsJSON-774334068 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:07.967318 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ba48ca0b-82e5-4dd4-a030-5a112a2b98be tempest-BackendsCapabilitiesAdminTestsJSON-774334068 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:11.313840 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Flow 'volume_create_scheduler' (589ee372-7dea-4140-ab69-c2a711deb00b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:11.317283 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c01d16ea-4179-45a0-9a95-b7ad1adb29c4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:11.320228 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c01d16ea-4179-45a0-9a95-b7ad1adb29c4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(deec2fe7-e868-4e58-ab59-21b047591bf9),volume_id=deec2fe7-e868-4e58-ab59-21b047591bf9,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:11.323229 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d046521b-bcfa-4073-b843-f552a015ac11) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:11.333461 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-67af4679-10dd-4496-90b8-9353e9ffe814 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:11.357987 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:11.358781 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:11.359277 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:11.360398 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (deec2fe7-e868-4e58-ab59-21b047591bf9) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:11.360902 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:06.193384 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:11.362438 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:11.362924 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:11.363392 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:06.193384] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:11.364056 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Weigher CapacityWeigher returned, weigher value is {max: 380.79999999999995, min: 380.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:11.364498 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:11.365187 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:11.365753 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:11.365215 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:11.397114 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d046521b-bcfa-4073-b843-f552a015ac11) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:11.402562 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1a8aed43-3dfc-4b11-84d3-924efdab8409 tempest-AttachVolumeTestJSON-1552858705 None] Flow 'volume_create_scheduler' (589ee372-7dea-4140-ab69-c2a711deb00b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:14.275099 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Flow 'volume_create_scheduler' (55005c41-8ae7-417f-b0d3-e5436baf6f01) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:14.277840 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9af8c8e1-f846-434d-8251-e39c960b8716) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:14.280733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9af8c8e1-f846-434d-8251-e39c960b8716) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5f2be858-8823-4023-9db9-0d81a705687b),volume_id=5f2be858-8823-4023-9db9-0d81a705687b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:14.283301 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bcb29a5e-9b82-4b71-8b2b-da09d02b474f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:14.292685 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:14.310894 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:14.311275 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:14.313345 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:14.313896 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5f2be858-8823-4023-9db9-0d81a705687b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:14.314363 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:11.365215 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:14.315448 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:14:14.317577 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:14.319598 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:14.321351 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:14.324899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:14.324899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:14.324899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:11.365215] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:14.324899 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:14.326298 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:14.326897 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:14.327369 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:14.326935 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:14.362417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bcb29a5e-9b82-4b71-8b2b-da09d02b474f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:14.364439 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-01711c36-2dc5-42f2-834f-eb5fc4c5f2b0 tempest-SnapshotsActionsTest-522976871 None] Flow 'volume_create_scheduler' (55005c41-8ae7-417f-b0d3-e5436baf6f01) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:15.517737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:15.517737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:15.517737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:15.517737 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:15.518620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:14.326935 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:15.519880 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:15.520465 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:15.520828 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:14.326935] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:15.521363 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:15.521647 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:15.525951 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ae62ef8b-ad88-4243-a28e-f91afa07d1ac tempest-SnapshotsActionsTest-522976871 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:15.525523 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:18.179669 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:18.180234 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:18.180466 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:18.180613 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:18.181175 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:15.525523 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:14:18.181692 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:18.183591 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:18.185588 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 326.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:18.190664 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:18.193462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:18.193462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:18.193462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:15.525523] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:18.193462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Weigher CapacityWeigher returned, weigher value is {max: 377.79999999999995, min: 377.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:18.193462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:18.193933 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3977beeb-48ac-4f85-9965-31a6a8aa8740 tempest-SnapshotsActionsTest-522976871 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.32, total_capacity_gb: 19.79,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:18.183979 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:20.824403 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:20.825109 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:20.825216 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:20.825582 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:20.826073 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.32, total_capacity_gb: 19.79,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:18.183979 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:20.827258 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:20.827610 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:20.828527 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.32, total_capacity_gb: 19.79,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:18.183979] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:20.829171 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Weigher CapacityWeigher returned, weigher value is {max: 376.79999999999995, min: 376.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:20.829520 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:20.829911 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ad008f12-3306-4c0d-a095-15fee162c5e6 tempest-SnapshotsActionsTest-522976871 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.32, total_capacity_gb: 19.79,allocated_capacity_gb: 20, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 20,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:20.829565 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:23.354463 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:23.355857 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:23.356567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:23.357141 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:23.357692 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.32, total_capacity_gb: 19.79,allocated_capacity_gb: 20, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 20,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:20.829565 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:23.358744 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 306.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:23.361876 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:23.364474 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 286.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:14:23.366548 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:14:23.369606 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:23.371807 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:23.371807 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:23.371807 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.32, total_capacity_gb: 19.79,allocated_capacity_gb: 20, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 20,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:20.829565] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:23.372553 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Weigher CapacityWeigher returned, weigher value is {max: 375.79999999999995, min: 375.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:23.373110 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:23.373736 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-96e2e760-ab3f-4a90-b976-7402f32c539f tempest-SnapshotsActionsTest-522976871 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.32, total_capacity_gb: 19.79,allocated_capacity_gb: 21, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 21,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:23.373355 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:28.047700 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 13, 40, 494914)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:28.048351 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:28.049455 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:28.050043 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:28.050630 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.32, total_capacity_gb: 19.79,allocated_capacity_gb: 21, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 21,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:23.373355 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:28.052055 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:28.052910 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:28.053467 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.32, total_capacity_gb: 19.79,allocated_capacity_gb: 21, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 21,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:23.373355] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:28.054209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Weigher CapacityWeigher returned, weigher value is {max: 374.79999999999995, min: 374.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:28.054865 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:28.055414 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-21f3f644-2704-4cc3-bd62-a41fd413c82b tempest-SnapshotsActionsTest-522976871 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 11.32, total_capacity_gb: 19.79,allocated_capacity_gb: 22, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 22,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:28.055066 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:33.579022 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0a37328d-bfc1-4adc-9be8-c6e2ceb88375 tempest-SnapshotsActionsTest-522976871 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:14:39.719028 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Flow 'volume_create_scheduler' (52463571-646d-403d-a006-de28a32f7e6f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:39.728481 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (dd052226-d1d0-4c59-a778-4a551c88ead3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:39.731508 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (dd052226-d1d0-4c59-a778-4a551c88ead3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5b5f645b-1372-4562-9da0-9e5946929813),volume_id=5b5f645b-1372-4562-9da0-9e5946929813,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:39.740029 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a71b0603-1281-49b6-b54a-13a781da6c23) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:39.752055 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:39.784595 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 14, 33, 568594)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:39.784988 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:39.785680 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:39.786237 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5b5f645b-1372-4562-9da0-9e5946929813) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:39.786792 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:33.568594 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 266.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:39.788013 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:39.790590 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 246.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:14:39.792812 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:14:39.795432 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:14:39.799804 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:14:39.799804 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:39.802134 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:39.802134 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:33.568594] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:39.802686 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:39.803229 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:39.805153 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:39.806710 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:39.805389 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:39.858311 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a71b0603-1281-49b6-b54a-13a781da6c23) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:39.866029 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9adbeed7-3f3b-4ed7-b3c0-50c2fe6a6653 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Flow 'volume_create_scheduler' (52463571-646d-403d-a006-de28a32f7e6f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:43.082901 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2a892a6c-05f8-423d-9057-6b2cd61b3786 tempest-VolumeQuotasNegativeTestJSON-358974153 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:14:43.786357 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-af7a63e3-a788-47df-839d-08f615348d89 tempest-AttachVolumeTestJSON-1552858705 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:14:44.795834 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bdcd88f7-56c1-42c6-98a3-f28f28b469b4 tempest-AttachVolumeTestJSON-1552858705 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:14:46.308767 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a5a83696-fb8d-43de-b913-2869286617ed tempest-AttachVolumeTestJSON-1552858705 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:14:51.456243 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Flow 'volume_create_scheduler' (cfb1b03c-6b70-4971-96b3-b93c8649976f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:51.463341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (301e6ce0-9923-4ec6-aadd-3e59af207cff) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:51.476128 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (301e6ce0-9923-4ec6-aadd-3e59af207cff) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cdf0b542-adc4-4ad4-b8a5-f31ef25ec8a4),volume_id=cdf0b542-adc4-4ad4-b8a5-f31ef25ec8a4,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:51.478807 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b3757ce4-4b32-4a15-b3fc-a210e179ec02) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:51.493628 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:51.515422 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 14, 46, 300714)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:51.516968 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:51.517519 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:51.518068 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (cdf0b542-adc4-4ad4-b8a5-f31ef25ec8a4) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:51.518696 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.79,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:46.300714 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:51.520068 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:14:51.521927 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:14:51.524202 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:51.526553 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:51.563987 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:51.567132 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:51.567704 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.79,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:46.300714] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:51.568350 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Weigher CapacityWeigher returned, weigher value is {max: 382.79999999999995, min: 382.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:51.568706 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:51.569184 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:51.569566 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:51.569215 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:51.618806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b3757ce4-4b32-4a15-b3fc-a210e179ec02) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:51.622166 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9fe17e4e-a1f6-4ca1-b8cb-c8c84c88fbee tempest-VolumesBackupsAdminTest-848605363 None] Flow 'volume_create_scheduler' (cfb1b03c-6b70-4971-96b3-b93c8649976f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:52.942040 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-5cb8bd10-56c9-47ea-8c34-4de7aaf6af38 tempest-VolumesBackupsAdminTest-848605363 None] create_backup in rpcapi backup_id 71496ad9-9d3b-4d2a-8169-6aa64504c878 {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:14:54.602166 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a44e27ef-fde0-497d-828d-d285fcef96ff None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:14:58.085406 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (df4a2feb-2e98-42c2-99bd-7454a3399936) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:58.089287 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7e84b260-3266-4f99-88b4-409848579b87) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:58.098280 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7e84b260-3266-4f99-88b4-409848579b87) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(105b7bca-8395-4cce-8f08-70455ce9f127),volume_id=105b7bca-8395-4cce-8f08-70455ce9f127,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:58.103077 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (84461cbd-5aef-4879-a95c-dfa4548211d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:58.128431 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:58.176206 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 14, 46, 300714)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:58.176561 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:58.178447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:58.180586 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (105b7bca-8395-4cce-8f08-70455ce9f127) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:58.181173 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:51.569215 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:58.184157 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:58.184732 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:58.185262 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:51.569215] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:58.185866 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 381.79999999999995, min: 381.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:58.186546 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:58.188526 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:58.190497 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:58.190127 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:58.265189 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (84461cbd-5aef-4879-a95c-dfa4548211d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:58.272891 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b8ddaccc-a60c-4aad-ae0f-8dc2c890af24 tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (df4a2feb-2e98-42c2-99bd-7454a3399936) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:59.698757 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 14, 46, 300714)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:59.698757 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:59.698757 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:59.699568 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:59.699568 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:58.190127 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:14:59.700409 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:14:59.703842 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:14:59.706232 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:14:59.708491 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:14:59.714094 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:59.714094 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:59.714094 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.79,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:58.190127] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:59.714094 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 380.79999999999995, min: 380.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:59.714560 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:59.714560 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a805d0c-ab94-488f-be46-d8397ed5be5b tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:59.713274 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:59.754904 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Flow 'volume_create_scheduler' (b386a1e1-8017-4fe2-89c7-c391bb0f7d23) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:14:59.759705 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f1dae7fa-1887-4a95-b8c5-f51982203b0d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:59.765381 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f1dae7fa-1887-4a95-b8c5-f51982203b0d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(eb35ab6f-4427-44bb-afc6-132727ac2e52),volume_id=eb35ab6f-4427-44bb-afc6-132727ac2e52,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:59.768392 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (440b0f65-59f3-4212-a179-976571f436e9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:14:59.776961 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:14:59.804634 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 14, 46, 300714)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:14:59.805116 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:14:59.805548 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:59.805978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (eb35ab6f-4427-44bb-afc6-132727ac2e52) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:14:59.806466 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:59.713274 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:14:59.808478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:59.809052 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:14:59.809597 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:59.713274] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:14:59.810390 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:14:59.810931 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:14:59.811626 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:14:59.812018 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.309999999999999, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:59.811653 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:14:59.846706 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (440b0f65-59f3-4212-a179-976571f436e9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:14:59.848868 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-17d21849-f5f8-4feb-a76e-32aaf18fbf73 tempest-VolumesBackupsAdminTest-848605363 None] Flow 'volume_create_scheduler' (b386a1e1-8017-4fe2-89c7-c391bb0f7d23) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:02.162166 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (ba213582-bb6c-4986-8584-19dbff52398b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:02.165490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7a2189f7-128b-4e7a-9cdc-8932a6d45337) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:02.169061 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7a2189f7-128b-4e7a-9cdc-8932a6d45337) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=7ff3dbf0-2af5-4e6d-8bd3-c8a8e2b45bec,source_replicaid=,source_volid=None,volume=Volume(f2f588bc-c580-45eb-b38d-14197fb43905),volume_id=f2f588bc-c580-45eb-b38d-14197fb43905,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:02.171890 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (30bf204d-fa89-4fa5-a7fb-c8726cd354d7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:02.182345 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:02.199690 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 14, 46, 300714)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:02.200348 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:02.200919 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:02.201478 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (f2f588bc-c580-45eb-b38d-14197fb43905) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:02.202161 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.309999999999999, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:59.811653 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:02.203002 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:02.205324 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:02.207878 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 326.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:02.210031 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:02.212195 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:02.212195 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:02.212195 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:02.212195 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:02.212841 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.309999999999999, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:14:59.811653] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:02.213159 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:02.213598 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:02.213957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:02.214691 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.309999999999999, total_capacity_gb: 19.79,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:02.213984 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:02.248493 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (30bf204d-fa89-4fa5-a7fb-c8726cd354d7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:02.251472 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-72bac761-dfd0-4aa5-b934-9b04401f23c6 tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (ba213582-bb6c-4986-8584-19dbff52398b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:02.745535 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5fec7cd1-290c-4b06-8040-8358df985ff0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 18, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:04.160307 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4509417c-d7bf-4814-a450-6383b13ad983 tempest-VolumesSnapshotTestJSON-1178343811 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:04.761447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (4b25fd23-5cf7-4be6-8f6a-fc572987912d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:04.766290 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1ba6b53e-064e-401c-ba87-ce231e2af589) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:04.769069 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1ba6b53e-064e-401c-ba87-ce231e2af589) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c5b4b4bf-c148-413b-9970-636b07628a97),volume_id=c5b4b4bf-c148-413b-9970-636b07628a97,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:04.771825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ea2e2a22-9c41-4a1e-841e-e76abddf89e5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:04.782951 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:04.798368 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 4, 154238)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:04.798991 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:04.799421 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:04.799848 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c5b4b4bf-c148-413b-9970-636b07628a97) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:04.800300 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:04.154238 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:04.801708 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:04.804228 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 306.2, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:04.806607 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:04.810436 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:04.814378 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:04.814378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:04.814378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:04.814378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:04.154238] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:04.814378 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:04.814892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:04.814892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:04.814892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:04.804075 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:04.831462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ea2e2a22-9c41-4a1e-841e-e76abddf89e5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:04.833618 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-130ef682-10d8-4b89-9d2f-78c6d15617fc tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (4b25fd23-5cf7-4be6-8f6a-fc572987912d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:05.476907 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9d3ef6cb-3a45-4b6d-9ff1-c9ca61b821ec tempest-VolumesBackupsAdminTest-848605363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:06.001860 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 5, 471373)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:06.002473 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:06.003202 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:06.004108 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:06.004852 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:05.471373 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:06.008381 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:06.008892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:06.010987 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.79,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:05.471373] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:06.011511 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:06.012020 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:06.012626 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5bfdaece-43a3-4545-98bb-adbbe0245348 tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:06.012140 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:06.679886 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Flow 'volume_create_scheduler' (7dc4d94a-d660-483a-a127-43b463f9bfce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:06.684246 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (30a49901-e062-420c-a7f1-2d48dfed2882) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:06.691734 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (30a49901-e062-420c-a7f1-2d48dfed2882) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2c5ebbcb-4d38-4a44-9a19-ea90342400f6),volume_id=2c5ebbcb-4d38-4a44-9a19-ea90342400f6,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:06.694149 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7af0d716-f7b1-40a6-acd9-09395626f871) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:06.704396 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:06.720573 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 5, 471373)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:06.721038 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:06.721466 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:06.721905 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2c5ebbcb-4d38-4a44-9a19-ea90342400f6) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:06.722391 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:06.012140 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:06.723561 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:06.725397 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:06.728965 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:06.732071 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:06.738831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:06.738831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:06.738831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.79,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:06.012140] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:06.738831 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Weigher CapacityWeigher returned, weigher value is {max: 378.79999999999995, min: 378.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:06.739209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:06.739209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:06.739209 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:06.727927 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:06.773220 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7af0d716-f7b1-40a6-acd9-09395626f871) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:06.775707 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-dcc046e7-a8c3-4b5e-8d89-f082e7b5e725 tempest-VolumesBackupsAdminTest-848605363 None] Flow 'volume_create_scheduler' (7dc4d94a-d660-483a-a127-43b463f9bfce) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:07.950866 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-222365dd-d095-4c52-9056-8934c267a811 tempest-VolumesBackupsAdminTest-848605363 None] create_backup in rpcapi backup_id 0a3ee2ae-e7ac-4326-b38d-87c0677871a3 {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:15:08.740237 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (89fd4665-3d7f-4590-952f-21e3a23efc59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:08.742806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (da0c7f99-b941-40ea-9617-4becb9efd66b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:08.753635 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (da0c7f99-b941-40ea-9617-4becb9efd66b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=dae6c0cc-6f89-41ad-81a9-0523b448ad2c,source_replicaid=,source_volid=None,volume=Volume(905c7a2c-5a76-4099-a555-6a5d8a109656),volume_id=905c7a2c-5a76-4099-a555-6a5d8a109656,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:08.757269 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (864b693c-7198-4de9-b696-df1db894a07f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:08.765952 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:08.783665 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 5, 471373)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:08.784469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:08.785057 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:08.785556 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (905c7a2c-5a76-4099-a555-6a5d8a109656) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:08.786266 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:06.727927 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:08.787622 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:08.789164 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:08.790858 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:08.792476 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:08.793868 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:08.796690 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:08.797470 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:08.798000 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.79,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:06.727927] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:08.799788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Weigher CapacityWeigher returned, weigher value is {max: 377.79999999999995, min: 377.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:08.800334 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:08.800849 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:08.803305 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.79,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:08.802897 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:08.897203 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (864b693c-7198-4de9-b696-df1db894a07f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:08.908897 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-18f143fb-7a51-44ba-9cf5-2e81aa7b3d1d tempest-VolumesSnapshotTestJSON-1178343811 None] Flow 'volume_create_scheduler' (89fd4665-3d7f-4590-952f-21e3a23efc59) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:11.127523 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0cb1def7-4fc3-4d76-8f40-5281a20540da tempest-VolumesSnapshotTestJSON-1178343811 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:13.397210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5913feb8-7dad-40a0-91f6-2051f9f84545 tempest-VolumesSnapshotTestJSON-1178343811 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 16, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:16.306447 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a8deb380-f806-403c-84bf-2077a8277987 tempest-VolumesBackupsAdminTest-848605363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:18.350621 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-562168db-f2b4-46c1-a4c3-06df53a14bcc tempest-VolumesBackupsAdminTest-848605363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:18.663037 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f2e0c53c-e453-426d-884a-7f41dbb58fc0 tempest-VolumesSnapshotTestJSON-1178343811 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:21.071141 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-52359dbb-43fe-475f-94bf-c742dce3ad63 tempest-VolumesSnapshotTestJSON-1178343811 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:25.173283 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a37b2401-4b78-487f-9337-2894a3e25013 tempest-VolumesSnapshotTestJSON-1178343811 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:25.505650 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Flow 'volume_create_scheduler' (6c15928d-09b1-479b-bc1e-5e3bbe728a54) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:25.522989 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f1a2acde-570d-4684-bbe2-4ae4f3b4b239) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:25.532978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f1a2acde-570d-4684-bbe2-4ae4f3b4b239) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0cf99ee6-dd4b-4c88-94a7-f575a099de96),volume_id=0cf99ee6-dd4b-4c88-94a7-f575a099de96,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:25.538367 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4dbe0f12-13de-437e-9350-2037b1f92db8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:25.581778 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:25.624794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 25, 168238)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:25.626323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:25.626781 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:25.627241 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0cf99ee6-dd4b-4c88-94a7-f575a099de96) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:25.628857 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:25.168238 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:25.632939 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:25.634562 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:25.636332 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:25.637809 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:25.640245 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:25.640245 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:25.641009 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:25.643880 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:25.645670 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:25.168238] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:25.649527 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Weigher CapacityWeigher returned, weigher value is {max: 384.6, min: 384.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:25.649968 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:25.650487 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:25.650885 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:25.650518 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:25.712477 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4dbe0f12-13de-437e-9350-2037b1f92db8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:25.716403 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-124794f1-65f4-44ca-b5a2-fb6355a21677 tempest-VolumesActionsTest-1676956077 None] Flow 'volume_create_scheduler' (6c15928d-09b1-479b-bc1e-5e3bbe728a54) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:29.143637 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Flow 'volume_create_scheduler' (2de7fe59-c17b-4812-a625-15596ba3e536) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:29.149264 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1cd15612-5721-4034-9dc2-d3bf719d47a8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:29.152212 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1cd15612-5721-4034-9dc2-d3bf719d47a8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(3ba25c2d-c1c9-4756-8b3f-8c5c44039b60),volume_id=3ba25c2d-c1c9-4756-8b3f-8c5c44039b60,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:29.154978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4f1243b8-bb6f-48f5-872b-99af4f6b9269) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:29.163548 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:29.181513 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 25, 168238)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:29.181972 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:29.182433 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:29.182987 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (3ba25c2d-c1c9-4756-8b3f-8c5c44039b60) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:29.183892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:25.650518 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:29.186161 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:29.186838 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:29.187670 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:25.650518] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:29.188283 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Weigher CapacityWeigher returned, weigher value is {max: 383.6, min: 383.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:29.189041 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:29.189583 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:29.190034 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:29.189617 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:29.227078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4f1243b8-bb6f-48f5-872b-99af4f6b9269) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:29.229808 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fcd14e94-81c6-4ba5-9a69-842c7255bdfd tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Flow 'volume_create_scheduler' (2de7fe59-c17b-4812-a625-15596ba3e536) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:30.599044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 25, 168238)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:30.599736 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:30.599836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:30.600114 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:30.603753 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:29.189617 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:15:30.605919 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:15:30.608063 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:15:30.609613 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:15:30.611147 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:15:30.616769 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:30.617294 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:30.618620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:29.189617] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:30.619813 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Weigher CapacityWeigher returned, weigher value is {max: 381.6, min: 381.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:30.624420 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:30.626270 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cbe88831-3ae0-492b-bfb7-bf8e0bcc72e2 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.309999999999999, total_capacity_gb: 19.78,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:30.625854 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:34.773577 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-08d58b3a-2ee1-45dd-9121-214d236514a6 tempest-VolumesSnapshotNegativeTestJSON-1691853596 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:15:54.606431 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-06527ddd-baf4-42df-b92c-4c59143d4573 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:15:59.316998 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (6d4beeac-3aaa-4e50-a989-23784f4a5a37) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:15:59.319614 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (77663c9c-4843-4279-8871-339f35716d74) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:59.329881 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (77663c9c-4843-4279-8871-339f35716d74) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(018c6ae3-8cc0-463f-a270-8d9be05e21cf),volume_id=018c6ae3-8cc0-463f-a270-8d9be05e21cf,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:59.334542 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7131046d-4f8d-4ad2-a54a-455c2f1ff7cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:15:59.345628 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:15:59.409929 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 34, 765778)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:15:59.411731 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:15:59.412781 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:59.413170 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (018c6ae3-8cc0-463f-a270-8d9be05e21cf) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:15:59.413586 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:34.765778 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:15:59.415684 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:59.416414 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:15:59.417098 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:34.765778] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:15:59.417849 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 383.6, min: 383.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:15:59.418448 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:15:59.420498 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:15:59.421059 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:59.420688 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:15:59.467571 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7131046d-4f8d-4ad2-a54a-455c2f1ff7cd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:15:59.470097 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-b4c6ca16-6df0-4a1b-9d89-3c388fd74886 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (6d4beeac-3aaa-4e50-a989-23784f4a5a37) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:01.837605 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 15, 34, 765778)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:01.838296 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:01.838531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:01.854147 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:01.858307 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:59.420688 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 326.2, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:01.859938 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:16:01.861537 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:16:01.863254 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:01.864508 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:01.865936 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:01.865936 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:01.882218 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:01.882541 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:01.882925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:15:59.420688] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:01.883333 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 382.6, min: 382.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:01.885490 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:01.886639 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bdc0a694-f409-43d3-8131-3ee7476fd051 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:01.886273 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:02.800787 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5386f632-1500-4201-b0c2-e22b0c2b7896 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:16:06.891613 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (e18ee001-d2c5-466a-b061-f46342bf0a4c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:06.894852 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2900cb31-a9ca-4ba4-8ff5-eb7dd7b7c74c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:06.898122 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2900cb31-a9ca-4ba4-8ff5-eb7dd7b7c74c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=addd3daf-d0c7-4498-8cb0-fefe0954833b,source_replicaid=,source_volid=None,volume=Volume(0660ccb1-0548-40f8-8d57-c0abfdeff803),volume_id=0660ccb1-0548-40f8-8d57-c0abfdeff803,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:06.900253 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f24cf539-2593-4753-90f7-fb8967b9dc84) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:06.909770 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:16:06.934112 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 16, 2, 795947)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:06.934733 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:06.935464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:06.935978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0660ccb1-0548-40f8-8d57-c0abfdeff803) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:06.936549 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:02.795947 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:06.937614 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:06.939396 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:16:06.940855 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:06.943428 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:06.947407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:06.947407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:06.947407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:02.795947] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:06.947407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 382.6, min: 382.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:06.948004 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:06.948004 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:16:06.948004 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:06.940445 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:06.991259 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f24cf539-2593-4753-90f7-fb8967b9dc84) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:06.998040 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-9d091edb-a9cf-4deb-b1a5-6356ba5c3b10 req-4af56a37-a6ad-4440-a17d-54a20b93d7eb tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (e18ee001-d2c5-466a-b061-f46342bf0a4c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:23.630161 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-66175c27-ffcb-4397-90b9-4eab84faaa6a req-b62e71c1-786e-4a51-a5ec-dd68c8b0a1a7 tempest-TestVolumeBootPattern-1052726415 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:16:26.979045 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-00e9a586-290a-43d6-8824-378f823ab7ef tempest-TestVolumeBootPattern-1052726415 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:16:27.906683 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (c6701f51-5919-4280-a1a7-489b8bb24e51) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:27.916366 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c2fb2f49-6cae-4e5b-9223-31f9a8ad0fa1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:27.932424 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c2fb2f49-6cae-4e5b-9223-31f9a8ad0fa1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c8ebc1cb-1b9f-4042-95e4-e9b0f9b55411),volume_id=c8ebc1cb-1b9f-4042-95e4-e9b0f9b55411,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:27.936382 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6bd40db4-5d4b-4913-aba1-1e90cf9fd386) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:27.955207 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:16:27.992956 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 16, 26, 974583)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:27.994035 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:27.994727 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:27.995241 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c8ebc1cb-1b9f-4042-95e4-e9b0f9b55411) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:27.995771 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:26.974583 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:27.997892 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:27.998471 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:27.998992 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:26.974583] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:27.999985 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 383.6, min: 383.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:28.003565 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:28.006580 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:16:28.009185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:28.007107 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:28.076154 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6bd40db4-5d4b-4913-aba1-1e90cf9fd386) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:28.100952 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-118bd8e1-1e55-48ce-8f4a-a3f6e05b9a93 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (c6701f51-5919-4280-a1a7-489b8bb24e51) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:37.969889 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Flow 'volume_create_scheduler' (0aebb541-5ed2-4b18-a075-28433cfc997b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:37.972703 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0a945287-b054-4913-9927-0bd10da31337) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:37.975864 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0a945287-b054-4913-9927-0bd10da31337) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f617d83e-21a6-44f4-89dc-18af33af315a),volume_id=f617d83e-21a6-44f4-89dc-18af33af315a,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:37.978032 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c7ee044b-5f55-442e-9b29-a612e0d243a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:37.986267 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:16:38.005175 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 16, 26, 974583)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:38.005984 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:38.006533 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:38.007204 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f617d83e-21a6-44f4-89dc-18af33af315a) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:38.007755 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:28.007107 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:38.009017 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:16:38.011122 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:38.012480 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:38.013809 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:38.016884 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:38.017720 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:38.018623 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:28.007107] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:38.019226 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Weigher CapacityWeigher returned, weigher value is {max: 382.6, min: 382.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:38.019829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:38.021020 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:16:38.021548 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:38.021190 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:38.075081 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c7ee044b-5f55-442e-9b29-a612e0d243a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:38.079391 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-8eeaa8d2-b215-49ba-b56e-de79f40e9960 tempest-AbsoluteLimitsTests-1414896416 None] Flow 'volume_create_scheduler' (0aebb541-5ed2-4b18-a075-28433cfc997b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:40.207762 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9d2e2e65-39a6-4e09-8e95-cdebda50645d tempest-AbsoluteLimitsTests-1414896416 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:16:43.863921 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ecc13e4e-f740-4f1b-b982-7e56a25cbf25 tempest-VolumesActionsTest-1676956077 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:16:46.851559 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 16, 43, 857035)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:46.852081 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:46.852370 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:46.852811 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:46.853219 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:43.857035 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:46.854462 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:46.855132 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:46.855690 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:43.857035] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:46.856318 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 383.6, min: 383.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:46.857863 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:46.858509 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-ab566bc2-ac5a-4fd2-9f8c-330742e9f925 req-f752da79-8cd7-4e46-a560-8211810e1ffb tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:46.858150 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:47.040888 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Flow 'volume_create_scheduler' (4abe3d66-f241-4023-a8df-3af003d74485) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:47.044840 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1680d1ab-6b1e-46bf-99d2-fe37831bade5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:47.051740 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1680d1ab-6b1e-46bf-99d2-fe37831bade5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7db9043d-ea67-4d40-b95e-09a5d3335c47),volume_id=7db9043d-ea67-4d40-b95e-09a5d3335c47,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:47.055168 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7f712330-cb9d-47cf-bc6a-611cd1ca4095) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:47.070556 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:16:47.099895 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 16, 43, 857035)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:47.100323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:47.100805 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:47.101195 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (7db9043d-ea67-4d40-b95e-09a5d3335c47) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:47.101591 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:46.858150 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:16:47.102212 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:47.103918 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:47.105651 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:47.107611 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:47.109211 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:47.109211 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:47.109211 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:46.858150] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:47.109211 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Weigher CapacityWeigher returned, weigher value is {max: 382.6, min: 382.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:47.109211 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:47.109517 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:16:47.109517 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:47.105546 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:47.145746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7f712330-cb9d-47cf-bc6a-611cd1ca4095) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:47.148188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fec9b448-e9da-4909-b672-93096d94a213 tempest-VolumesBackupsTest-964287589 None] Flow 'volume_create_scheduler' (4abe3d66-f241-4023-a8df-3af003d74485) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:51.065850 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (8a9bc164-0239-4d08-8cc2-56c0a1f8b67e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:51.068341 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (04538fe0-ccf7-4b40-82a1-f9238b493683) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:51.076944 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (04538fe0-ccf7-4b40-82a1-f9238b493683) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=bdcca7d9-0af7-4910-9c78-35b3d732f3aa,source_replicaid=,source_volid=None,volume=Volume(56a4462b-d3c7-4f86-bf33-f3e252d4c4cb),volume_id=56a4462b-d3c7-4f86-bf33-f3e252d4c4cb,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:51.091947 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8bb85e2f-3888-4439-a810-384318a7ee03) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:16:51.101749 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:16:51.132677 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 16, 43, 857035)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:16:51.133372 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:16:51.133942 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:51.134541 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (56a4462b-d3c7-4f86-bf33-f3e252d4c4cb) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:16:51.136041 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:47.105546 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:16:51.137754 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:51.139398 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:16:51.140727 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:16:51.142137 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:16:51.143751 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:16:51.143751 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:16:51.146007 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:51.146682 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:16:51.147400 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:47.105546] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:16:51.148053 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 381.6, min: 381.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:16:51.148602 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:16:51.149183 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:16:51.149768 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.78,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:16:51.149414 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:16:51.205321 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8bb85e2f-3888-4439-a810-384318a7ee03) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:16:51.213018 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [req-c5e54fc6-e17d-4a59-a2bb-1ddfcc20c047 req-23d5c514-ed0e-4853-9bf3-ebeb14127e60 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (8a9bc164-0239-4d08-8cc2-56c0a1f8b67e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:16:54.618772 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fb16940f-9fd2-41d6-8f5f-6b1f5dea747d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:17:02.819388 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-939efabe-a2ef-438d-8c86-4ee42edef6b5 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:17:09.203958 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-70757f63-e9fa-4d12-a7bd-7477bd4f03b8 req-fe8ecc44-a3d4-4c11-b41e-60023e92444b tempest-TestVolumeBootPattern-1052726415 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:17:11.200451 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-62ce7bac-e971-4296-a961-f41b2c7c48ba tempest-VolumesBackupsTest-964287589 None] create_backup in rpcapi backup_id dda0613a-e4a9-4210-b0c1-418b090a1872 {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:17:22.419882 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-27003615-ac15-44cc-b538-882c2d786190 tempest-VolumesBackupsTest-964287589 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:17:23.236321 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Flow 'volume_create_scheduler' (0d38c337-ca6d-4c22-baf4-c5519422e369) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:23.242206 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (baf39657-d5e3-4544-968b-d4b3d76ea495) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:23.248415 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (baf39657-d5e3-4544-968b-d4b3d76ea495) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(63f35122-865d-4f8c-aea6-9339ec61e273),volume_id=63f35122-865d-4f8c-aea6-9339ec61e273,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:23.256590 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c1f7f351-3964-4f29-ba48-afd25415a2dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:23.270529 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:17:23.297324 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 22, 410647)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:17:23.298294 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:17:23.298957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:23.299586 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (63f35122-865d-4f8c-aea6-9339ec61e273) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:17:23.300278 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:22.410647 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:17:23.302085 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 346.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:17:23.305529 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:17:23.324877 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:17:23.330934 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:17:23.335954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:23.335954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:23.335954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.78,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:22.410647] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:17:23.335954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Weigher CapacityWeigher returned, weigher value is {max: 383.6, min: 383.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:17:23.335954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:17:23.337131 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:17:23.337131 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:23.315821 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:17:23.357344 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c1f7f351-3964-4f29-ba48-afd25415a2dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:23.361159 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8c91845-8272-4beb-a958-aa199627cfba tempest-VolumesBackupsTest-964287589 None] Flow 'volume_create_scheduler' (0d38c337-ca6d-4c22-baf4-c5519422e369) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:25.721766 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-27bd7145-3d7d-433c-9df5-f6c6856cb4d8 tempest-VolumesBackupsTest-964287589 None] create_backup in rpcapi backup_id 2284c44a-c268-439c-ba75-802eae09a4da {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:17:36.467031 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Flow 'volume_create_scheduler' (5976343b-1b6a-47b3-8d05-96e1dc4ee2f6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:36.471124 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a1077b1a-2512-4739-9af0-09f9c990c9b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:36.475064 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a1077b1a-2512-4739-9af0-09f9c990c9b1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4c3ee60c-9ec2-4b45-93c7-8f09c20e3df1),volume_id=4c3ee60c-9ec2-4b45-93c7-8f09c20e3df1,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:36.478401 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f020c273-b7db-419f-9b43-08c54f4c392a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:36.490174 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:17:36.515137 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 22, 410647)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:17:36.515975 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:17:36.516662 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:36.516912 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4c3ee60c-9ec2-4b45-93c7-8f09c20e3df1) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:17:36.518210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:23.315821 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:17:36.519409 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:36.520111 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:36.520745 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.78,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:23.315821] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:17:36.521620 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Weigher CapacityWeigher returned, weigher value is {max: 382.6, min: 382.6} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:17:36.522223 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:17:36.522998 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:17:36.523547 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.27, total_capacity_gb: 19.78,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:36.523168 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:17:36.560510 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f020c273-b7db-419f-9b43-08c54f4c392a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:36.562925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-439efcf9-756b-4b62-a1df-68ffeb806ebc tempest-VolumesBackupsTest-964287589 None] Flow 'volume_create_scheduler' (5976343b-1b6a-47b3-8d05-96e1dc4ee2f6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:46.206089 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-ed73d6b0-49df-4692-99e9-710775fdd2ca req-b071f7f3-0a52-418a-8956-55813820babb tempest-TestVolumeBootPattern-1052726415 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.3, 'free_capacity_gb': 18.38, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:17:47.017553 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-604b9235-5c4e-4088-a48e-4030085498b8 tempest-VolumesBackupsTest-964287589 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.3, 'free_capacity_gb': 18.38, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:17:48.505330 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (e73d6d39-3b5e-4219-917c-cf35a450fd8d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:48.507925 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8338793d-d03e-496a-8dcf-1eb86b217a42) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:48.512113 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8338793d-d03e-496a-8dcf-1eb86b217a42) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(1c525edb-e9c4-4a6d-98f9-a951189a0927),volume_id=1c525edb-e9c4-4a6d-98f9-a951189a0927,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:48.514180 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ac83f0f2-d5be-4190-8273-2f239695d622) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:48.522281 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:17:48.543163 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.3, 'free_capacity_gb': 18.38, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 47, 13140)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:17:48.543663 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:17:48.544058 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:48.545567 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1c525edb-e9c4-4a6d-98f9-a951189a0927) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:17:48.546221 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.38, total_capacity_gb: 19.3,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:47.013140 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:17:48.546862 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:17:48.548505 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:17:48.550389 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:17:48.551764 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:17:48.555258 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:48.555969 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:48.556388 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.38, total_capacity_gb: 19.3,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:47.013140] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:17:48.556799 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 374.0, min: 374.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:17:48.557210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:17:48.557602 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:17:48.557982 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.38, total_capacity_gb: 19.3,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:48.557630 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:17:48.597600 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ac83f0f2-d5be-4190-8273-2f239695d622) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:48.600755 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-700052ef-1e7c-4678-a8ff-cee11b82aa93 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (e73d6d39-3b5e-4219-917c-cf35a450fd8d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:52.368634 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ebbf8217-4e1c-4358-8a77-4b670c05dace tempest-VolumesBackupsTest-964287589 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:17:54.610860 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d9465222-fb30-4f50-9182-9e81d71e6c8c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:17:55.237380 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (9a394677-2d6d-4d9d-b70d-244fef147ba4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:55.251667 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b91a3955-eeb5-4a94-90b1-f13eb29d32e3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:55.261126 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b91a3955-eeb5-4a94-90b1-f13eb29d32e3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e419ff84-0945-4744-8dfa-173b5480e122),volume_id=e419ff84-0945-4744-8dfa-173b5480e122,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:55.265574 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c258b506-ee86-4d0b-92ed-378a5baedaa4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:55.281519 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:17:55.319937 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 52, 360954)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:17:55.320626 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:17:55.321135 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:55.321781 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e419ff84-0945-4744-8dfa-173b5480e122) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:17:55.323100 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.74,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:52.360954 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:17:55.324629 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:17:55.327231 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 367.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:17:55.328790 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:17:55.330502 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:17:55.332531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:55.332531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:55.332531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.74,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:52.360954] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:17:55.332531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Weigher CapacityWeigher returned, weigher value is {max: 382.79999999999995, min: 382.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:17:55.332531 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:17:55.332903 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:17:55.332903 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.74,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:55.331154 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:17:55.390365 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c258b506-ee86-4d0b-92ed-378a5baedaa4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:55.392562 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9971a971-2e2b-462c-a035-b5e1d1475132 tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (9a394677-2d6d-4d9d-b70d-244fef147ba4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:56.659289 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (8c885249-9095-4e21-9ebc-83a85ef3ad56) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:56.662363 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (99d6c011-f2e0-41d1-81e2-f1eb882f36a9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:56.666016 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (99d6c011-f2e0-41d1-81e2-f1eb882f36a9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=None,source_replicaid=,source_volid=e419ff84-0945-4744-8dfa-173b5480e122,volume=Volume(9e0f769b-df7f-427b-ba55-52aab6436b1b),volume_id=9e0f769b-df7f-427b-ba55-52aab6436b1b,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:56.669195 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8a260fa2-2ac1-411f-b2ce-ced07abcc75f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:56.677880 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:17:56.696491 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 52, 360954)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:17:56.697510 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:17:56.698360 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:56.699204 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9e0f769b-df7f-427b-ba55-52aab6436b1b) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:17:56.700500 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.74,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:55.331154 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:17:56.701357 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:17:56.703018 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:17:56.704641 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:17:56.707659 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:17:56.710836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:56.710836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:56.710836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.74,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:55.331154] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:17:56.710836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Weigher CapacityWeigher returned, weigher value is {max: 381.79999999999995, min: 381.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:17:56.710836 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:17:56.711495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:17:56.711495 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.27, total_capacity_gb: 19.74,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:56.703658 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:17:56.737560 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8a260fa2-2ac1-411f-b2ce-ced07abcc75f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:56.739788 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1ef2f361-b7c0-415c-8a55-01fadc7e389a tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (8c885249-9095-4e21-9ebc-83a85ef3ad56) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:59.133536 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (b17b9753-5821-4a92-9135-d06cf1820703) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:17:59.137312 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a84f19cc-01d8-49bf-94e4-231699edbeb5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:59.141380 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a84f19cc-01d8-49bf-94e4-231699edbeb5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(cbec42b6-f144-4b8a-9820-5d351d27b3ab),volume_id=cbec42b6-f144-4b8a-9820-5d351d27b3ab,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:59.143400 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8925226e-32fb-4424-9189-6fbf5d1cffdd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:17:59.152163 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:17:59.172953 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 52, 360954)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:17:59.173356 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:17:59.173723 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:59.174124 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (cbec42b6-f144-4b8a-9820-5d351d27b3ab) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:17:59.179472 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.27, total_capacity_gb: 19.74,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:56.703658 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:17:59.181957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:59.182546 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:17:59.183101 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.27, total_capacity_gb: 19.74,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:56.703658] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:17:59.183692 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Weigher CapacityWeigher returned, weigher value is {max: 380.79999999999995, min: 380.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:17:59.184208 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:17:59.184829 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:17:59.185369 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.27, total_capacity_gb: 19.74,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:59.185013 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:17:59.242035 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8925226e-32fb-4424-9189-6fbf5d1cffdd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:17:59.248268 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-bcf141ac-93d0-4209-bb4c-22b338a45368 tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (b17b9753-5821-4a92-9135-d06cf1820703) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:00.587152 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (01506e18-03c7-4ed3-a17c-97f612bba0ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:00.591273 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b62b0356-974b-4ac8-9dca-dd2aa464f4d5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:00.594224 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b62b0356-974b-4ac8-9dca-dd2aa464f4d5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=None,source_replicaid=,source_volid=cbec42b6-f144-4b8a-9820-5d351d27b3ab,volume=Volume(f8e273e4-8bb7-4888-a02d-ff82710713b2),volume_id=f8e273e4-8bb7-4888-a02d-ff82710713b2,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:00.596383 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5ad6f73d-5dff-40fb-9260-0259c074fb4b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:00.606934 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:18:00.629021 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 17, 52, 360954)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:18:00.629755 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:18:00.631071 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:00.631664 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (f8e273e4-8bb7-4888-a02d-ff82710713b2) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:18:00.632225 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.27, total_capacity_gb: 19.74,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:59.185013 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:00.632925 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:00.635113 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:18:00.638152 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 345.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:00.641832 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:00.648574 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:00.648574 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:00.649228 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.27, total_capacity_gb: 19.74,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:17:59.185013] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:18:00.650560 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Weigher CapacityWeigher returned, weigher value is {max: 379.79999999999995, min: 379.79999999999995} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:18:00.651775 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:18:00.652407 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:18:00.653159 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.27, total_capacity_gb: 19.74,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:00.652613 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:18:00.689269 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5ad6f73d-5dff-40fb-9260-0259c074fb4b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:00.692080 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-16ad28e9-e79a-424d-b5f9-b9e2edeb9c42 tempest-VolumesCloneTest-1255598884 None] Flow 'volume_create_scheduler' (01506e18-03c7-4ed3-a17c-97f612bba0ec) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:03.182978 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-13aa7ce3-4989-45ca-ab7b-3af5152839dd None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 17, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:18:03.404461 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4fd280b7-af1d-4f89-87a3-fc73f25e37ec tempest-VolumesCloneTest-1255598884 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:18:04.503044 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7e058a22-f2e7-406f-a036-167b6d9bd622 tempest-VolumesCloneTest-1255598884 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:18:06.477241 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-857964d1-e20b-4ae6-a7b2-19c3adbd22f5 tempest-VolumesCloneTest-1255598884 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:18:07.347714 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3ed7b885-c195-4beb-a727-c4874ae1698a tempest-VolumesCloneTest-1255598884 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:18:21.002343 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Flow 'volume_create_scheduler' (8e67562c-700a-4654-96fc-624310927f67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:21.004794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f565affd-3a11-4934-82d1-ed70904e0443) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:21.007548 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f565affd-3a11-4934-82d1-ed70904e0443) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(80cd86d6-7469-4707-b035-95103024f27c),volume_id=80cd86d6-7469-4707-b035-95103024f27c,volume_properties=VolumeProperties,volume_type=VolumeType(46d584c6-0769-4b5d-9fc1-b2805f95ce87))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:21.010210 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1c8132bb-bd29-4471-bf93-a0dbd2c398bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:21.017625 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:18:21.035933 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 18, 7, 341103)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:18:21.036350 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:18:21.036712 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:21.037069 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (80cd86d6-7469-4707-b035-95103024f27c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:18:21.037464 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:07.341103 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:21.038536 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:18:21.041251 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 325.4, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:21.045841 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:21.049272 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:21.051780 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:21.051780 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Volume type extra spec requirement "storage_protocol=tempest-storage_protocol-577128506" does not match reported capability "ceph" {{(pid=133499) _satisfies_extra_specs /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:86}} Nov 20 21:18:21.051780 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:07.341103 fails resource_type extra_specs requirements {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:100}} Nov 20 21:18:21.051780 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Filter CapabilitiesFilter returned 0 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:21.052104 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Filtering removed all hosts for the request with volume ID '80cd86d6-7469-4707-b035-95103024f27c'. Filter results: [('AvailabilityZoneFilter', ['ncdv-master-1@ceph#ceph']), ('CapacityFilter', ['ncdv-master-1@ceph#ceph']), ('CapabilitiesFilter', [])] {{(pid=133499) _log_filtration /opt/stack/cinder/cinder/scheduler/base_filter.py:73}} Nov 20 21:18:21.052104 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.scheduler.base_filter [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Filtering removed all hosts for the request with volume ID '80cd86d6-7469-4707-b035-95103024f27c'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 1), CapabilitiesFilter: (start: 1, end: 0) Nov 20 21:18:21.052104 ncdv-master-1 cinder-scheduler[133499]: WARNING cinder.scheduler.filter_scheduler [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] No weighed backend found for volume with properties: {'id': '46d584c6-0769-4b5d-9fc1-b2805f95ce87', 'name': 'tempest-UserMessagesTest-volume-type-775900055', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'tempest-storage_protocol-577128506', 'vendor_name': 'tempest-vendor_name-365242864'}, 'qos_specs_id': None, 'created_at': '2021-11-20T21:18:20.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False} Nov 20 21:18:21.052104 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.message.api [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Creating message record for request_id = req-3f4c5a8a-ec4a-458d-a152-be662bcef398 Nov 20 21:18:21.058293 ncdv-master-1 cinder-scheduler[133499]: ERROR cinder.scheduler.flows.create_volume [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Nov 20 21:18:21.058687 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.volume.flows.common [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Setting Volume 80cd86d6-7469-4707-b035-95103024f27c to error due to: No valid backend was found. No weighed backends available {{(pid=133499) error_out /opt/stack/cinder/cinder/volume/flows/common.py:81}} Nov 20 21:18:21.069848 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1c8132bb-bd29-4471-bf93-a0dbd2c398bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:21.072693 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-3f4c5a8a-ec4a-458d-a152-be662bcef398 tempest-UserMessagesTest-620810205 None] Flow 'volume_create_scheduler' (8e67562c-700a-4654-96fc-624310927f67) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:22.643814 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Flow 'volume_create_scheduler' (bc49b68e-60da-441f-aba9-a70d6815c1bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:22.647435 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fbde351f-d5a4-4b4b-bd16-d8dab65ec0b4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:22.650524 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fbde351f-d5a4-4b4b-bd16-d8dab65ec0b4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f814284a-474b-48ce-8d8e-85a81f4db17c),volume_id=f814284a-474b-48ce-8d8e-85a81f4db17c,volume_properties=VolumeProperties,volume_type=VolumeType(2d167704-672f-4454-829b-0496f3c38758))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:22.652702 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9f04d7d7-f0ea-4bbd-bd51-968652c6a583) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:22.660504 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:18:22.676553 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 18, 7, 341103)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:18:22.677189 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:18:22.677587 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:22.678014 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f814284a-474b-48ce-8d8e-85a81f4db17c) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:18:22.678508 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:07.341103 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:18:22.679645 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:22.682660 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:22.684546 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:18:22.687109 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:18:22.689765 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:22.689765 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Volume type extra spec requirement "storage_protocol=tempest-storage_protocol-1817706006" does not match reported capability "ceph" {{(pid=133499) _satisfies_extra_specs /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:86}} Nov 20 21:18:22.689765 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:07.341103 fails resource_type extra_specs requirements {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:100}} Nov 20 21:18:22.689765 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Filter CapabilitiesFilter returned 0 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:22.690128 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Filtering removed all hosts for the request with volume ID 'f814284a-474b-48ce-8d8e-85a81f4db17c'. Filter results: [('AvailabilityZoneFilter', ['ncdv-master-1@ceph#ceph']), ('CapacityFilter', ['ncdv-master-1@ceph#ceph']), ('CapabilitiesFilter', [])] {{(pid=133499) _log_filtration /opt/stack/cinder/cinder/scheduler/base_filter.py:73}} Nov 20 21:18:22.690128 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.scheduler.base_filter [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Filtering removed all hosts for the request with volume ID 'f814284a-474b-48ce-8d8e-85a81f4db17c'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 1), CapabilitiesFilter: (start: 1, end: 0) Nov 20 21:18:22.690128 ncdv-master-1 cinder-scheduler[133499]: WARNING cinder.scheduler.filter_scheduler [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] No weighed backend found for volume with properties: {'id': '2d167704-672f-4454-829b-0496f3c38758', 'name': 'tempest-UserMessagesTest-volume-type-1280991298', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'tempest-storage_protocol-1817706006', 'vendor_name': 'tempest-vendor_name-67727718'}, 'qos_specs_id': None, 'created_at': '2021-11-20T21:18:22.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False} Nov 20 21:18:22.690128 ncdv-master-1 cinder-scheduler[133499]: INFO cinder.message.api [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Creating message record for request_id = req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 Nov 20 21:18:22.692245 ncdv-master-1 cinder-scheduler[133499]: ERROR cinder.scheduler.flows.create_volume [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Nov 20 21:18:22.692678 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.volume.flows.common [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Setting Volume f814284a-474b-48ce-8d8e-85a81f4db17c to error due to: No valid backend was found. No weighed backends available {{(pid=133499) error_out /opt/stack/cinder/cinder/volume/flows/common.py:81}} Nov 20 21:18:22.704184 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9f04d7d7-f0ea-4bbd-bd51-968652c6a583) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:22.706461 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-992d25bb-bcd1-467d-b667-ee2ccd1e0d50 tempest-UserMessagesTest-620810205 None] Flow 'volume_create_scheduler' (bc49b68e-60da-441f-aba9-a70d6815c1bc) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:30.399401 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Flow 'volume_create_scheduler' (3f3d7bd7-5444-4e6c-a9fa-3e67da70f7d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:30.403297 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1a0c53af-585a-4826-ab5c-2964a6bc9515) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:30.407304 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1a0c53af-585a-4826-ab5c-2964a6bc9515) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f4b56d10-2e30-443a-b954-4dd7d242c0ad),volume_id=f4b56d10-2e30-443a-b954-4dd7d242c0ad,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:30.410494 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2b6273ae-1f0f-4c2c-8614-9fa883b69a7d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:30.421120 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:18:30.443183 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 18, 7, 341103)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:18:30.443657 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:18:30.443952 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:30.444317 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f4b56d10-2e30-443a-b954-4dd7d242c0ad) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:18:30.444710 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:07.341103 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:18:30.446230 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:30.446633 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:30.447342 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.27, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:07.341103] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:18:30.447571 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Weigher CapacityWeigher returned, weigher value is {max: 383.0, min: 383.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:18:30.448077 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:18:30.448698 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:18:30.449227 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.75,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:30.448873 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:18:30.499043 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2b6273ae-1f0f-4c2c-8614-9fa883b69a7d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:30.502386 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-08eac8fe-f254-4b5a-8cbd-fefba9481e19 tempest-VolumesBackupsV39Test-1666588577 None] Flow 'volume_create_scheduler' (3f3d7bd7-5444-4e6c-a9fa-3e67da70f7d8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:31.659213 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-297a450c-c0d8-41fa-88c0-76a2a36375eb tempest-VolumesBackupsV39Test-1666588577 None] create_backup in rpcapi backup_id edf623ef-7225-4148-8e74-88cafffed96f {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:18:39.103877 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e255bb94-7cf2-4e5e-b3f1-ddc5b22c6f5a tempest-VolumesBackupsV39Test-1666588577 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:18:51.604637 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Flow 'volume_create_scheduler' (1ad0e2b7-e2e5-4bb6-ada3-df16ac22517d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:51.610532 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cd017817-9f38-4127-8bbd-df3c732017d9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:51.614785 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cd017817-9f38-4127-8bbd-df3c732017d9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(14992743-5006-407a-ba5a-7bd61fae90ce),volume_id=14992743-5006-407a-ba5a-7bd61fae90ce,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:51.617972 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ff821011-0fad-4ca1-87e8-513bc80e1bbd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:51.629800 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:18:51.656070 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 18, 39, 99333)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:18:51.656070 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:18:51.656070 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:51.656070 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (14992743-5006-407a-ba5a-7bd61fae90ce) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:39.099333 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:51.656507 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:51.657971 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:18:51.660600 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:51.663547 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:51.665244 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.75,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:39.099333] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:18:51.666169 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Weigher CapacityWeigher returned, weigher value is {max: 383.0, min: 383.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:18:51.666169 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:18:51.666169 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:18:51.666169 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.75,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:51.651253 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:18:51.686428 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ff821011-0fad-4ca1-87e8-513bc80e1bbd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:51.689043 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-fae0c072-bf93-48ae-8933-19f3f7824f1b tempest-TestVolumeBackupRestore-667839818 None] Flow 'volume_create_scheduler' (1ad0e2b7-e2e5-4bb6-ada3-df16ac22517d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:54.002907 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.backup.rpcapi [None req-22720ad8-cf9a-4c4b-9e28-d6037846d6f7 tempest-TestVolumeBackupRestore-667839818 None] create_backup in rpcapi backup_id e2d330db-5233-464f-a223-c1b9124b8644 {{(pid=133499) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Nov 20 21:18:54.607257 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d23b780e-314f-46fa-ba24-7433f31a0319 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:18:59.474340 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Flow 'volume_create_scheduler' (738bda88-18d2-4beb-8c9e-ebe8e989c82b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:18:59.477379 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0b8eda8f-196c-46ad-9189-667d947b31b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:59.480384 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0b8eda8f-196c-46ad-9189-667d947b31b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(534ec160-6718-4124-90f8-da2967763760),volume_id=534ec160-6718-4124-90f8-da2967763760,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:59.483237 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (01945ce4-8bf8-4fa6-871c-6450d7d40e86) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:18:59.491835 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:18:59.509957 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 18, 39, 99333)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:18:59.510468 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:18:59.510834 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:59.511256 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (534ec160-6718-4124-90f8-da2967763760) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:18:59.511816 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.75,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:51.651253 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:59.513646 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:18:59.515865 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:18:59.517316 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:18:59.519247 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:18:59.521418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:59.521418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:18:59.521418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.75,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:51.651253] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:18:59.521418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Weigher CapacityWeigher returned, weigher value is {max: 382.0, min: 382.0} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:18:59.521418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:18:59.521926 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:18:59.521926 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.75,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:18:59.516569 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:18:59.546125 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (01945ce4-8bf8-4fa6-871c-6450d7d40e86) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:18:59.548307 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-0a6d2edd-991f-443f-9ad6-9467958fdcae tempest-TestVolumeBackupRestore-667839818 None] Flow 'volume_create_scheduler' (738bda88-18d2-4beb-8c9e-ebe8e989c82b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:19:03.234806 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0ce9f554-c50a-4db4-be40-7c9cbd07cbf9 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.28, 'free_capacity_gb': 18.73, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:19:54.604942 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-036de0bb-76da-40fd-b617-9e2d29b11dd3 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:20:03.224185 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b29bbe6e-a351-4de1-93bd-1a0cd59a6a2a None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.25, 'free_capacity_gb': 18.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:20:14.032883 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2fcd9857-2345-412d-a170-a84b0da67d8e tempest-TestVolumeBackupRestore-667839818 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.24, 'free_capacity_gb': 18.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:20:15.208931 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7ff26de1-ae8c-4189-be89-b1904c95d03d tempest-TestVolumeBackupRestore-667839818 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.24, 'free_capacity_gb': 18.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:20:54.603468 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-69694e82-20ad-4f38-ae2f-d3e93f939a50 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:21:03.141421 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6b48bf69-6269-4ab2-968d-aa3559d7ebef None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.67, 'free_capacity_gb': 19.11, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:21:54.598158 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3f6f5cee-2ebe-43e2-9972-6c433b633799 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:22:03.227954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b8ca506f-5741-43e5-8f69-e785e2095031 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.66, 'free_capacity_gb': 19.08, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:22:21.168744 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.66, 'free_capacity_gb': 19.08, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 22, 3, 214474)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:22:21.169449 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:22:21.170093 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:22:21.170093 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:22:21.170545 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.08, total_capacity_gb: 19.66,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:22:03.214474 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:22:21.171387 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:22:21.171746 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:22:21.172114 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.08, total_capacity_gb: 19.66,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:22:03.214474] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:22:21.172520 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 381.2, min: 381.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:22:21.172863 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:22:21.173251 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ecd77ac1-30a7-430f-8eb4-e03478f48f22 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.08, total_capacity_gb: 19.66,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:22:21.172907 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:22:22.512691 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (e63c87e9-eb6d-46cf-be71-d5f46adc6ca3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:22:22.515232 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (94cea494-ec4e-478c-8af4-846a079fd7df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:22:22.517794 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (94cea494-ec4e-478c-8af4-846a079fd7df) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=9d370d1c-6125-4777-a3d8-a2a3113463a0,source_replicaid=,source_volid=None,volume=Volume(da0d5896-ddcd-458d-a28f-29c7f2980025),volume_id=da0d5896-ddcd-458d-a28f-29c7f2980025,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:22:22.520456 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2ceb304e-283d-4469-92c7-631662778242) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:22:22.527828 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:22:22.542085 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.66, 'free_capacity_gb': 19.08, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 22, 3, 214474)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:22:22.542476 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:22:22.542833 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:22:22.543182 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (da0d5896-ddcd-458d-a28f-29c7f2980025) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:22:22.543566 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.08, total_capacity_gb: 19.66,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:22:21.172907 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:22:22.544582 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:22:22.546247 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: tgt_backend = self.driver.backend_passes_filters( Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:22:22.547849 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 381.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:22:22.549527 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:22:22.551323 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:22:22.551323 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:22:22.551323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:22:22.551323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:22:22.551323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.08, total_capacity_gb: 19.66,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:22:21.172907] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:22:22.551323 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Weigher CapacityWeigher returned, weigher value is {max: 380.2, min: 380.2} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:22:22.551729 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:22:22.551729 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:22:22.551729 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.08, total_capacity_gb: 19.66,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:22:22.546927 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:22:22.573817 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2ceb304e-283d-4469-92c7-631662778242) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:22:22.575886 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-9a0f165d-dfbb-4093-b876-2dde16a40d41 tempest-TestVolumeBootPattern-1052726415 None] Flow 'volume_create_scheduler' (e63c87e9-eb6d-46cf-be71-d5f46adc6ca3) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:22:54.608406 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1e7b27bc-40fe-447e-8ab5-79ec3fead442 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:23:03.155529 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-6aa1aa21-94b9-4783-914b-24bc43c1811e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.02, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:23:54.591514 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-678dfeea-17a2-4688-9f0d-b948d6e55ec0 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:24:03.146645 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fab2c31a-4462-45ae-a413-26329aa3944f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 18.98, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:24:24.534164 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-de1534ac-76b1-42bd-8e73-1d0682e0cc02 tempest-TestVolumeBootPattern-1052726415 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 18.98, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:24:54.598284 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d8bf8dad-6042-47cd-9fe0-3cff6abb6ac8 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:25:02.974687 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d18e192e-460d-4341-b820-29c3f6a0fa6f tempest-TestVolumeBootPattern-1052726415 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.06, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:25:03.167689 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dd617cc1-15aa-4320-939a-30dbdc403f6f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.06, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:25:54.594011 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-29b234af-29dc-4f5a-b6b1-69f57c79ef39 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:26:03.146449 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-37e86f33-12d3-4945-8d93-c32a44baed0c None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:26:54.601061 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d9dc82d1-6656-487b-b9f6-4c88ca00fd1a None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:27:03.151934 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1cf168d5-177c-4323-aaef-7ad7f5fadbb1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:27:54.605262 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4056ac7b-e021-470d-8233-d8e8f9a4c443 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:28:03.150771 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-507de2d9-82dd-4c75-b3c4-30d4f19121ae None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:28:54.610482 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-68be1dba-a860-458f-bdcf-b2c79aa18fce None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:29:03.150715 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8f87da48-df8e-4ae6-b1af-9ca1ae87b4f4 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:29:54.616119 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-ea334989-fd4c-4247-93fd-e3ff339cad02 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:30:03.150622 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2ca2c014-591e-4d0e-bca7-80c9df6b0dc1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:30:54.612750 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c674bfc2-e4a7-445d-8609-c5491d6a9a67 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:31:03.150248 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4650f120-df93-49b8-973b-3934ddaebde3 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:31:54.618963 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7061eaab-43d4-43b4-bd0a-4eb8ac956569 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:32:03.148701 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fb536086-3c1e-45ea-94d4-1d9065abe099 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:32:54.615702 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4e4d4a35-bb26-44ab-869d-416ffe359efb None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:33:03.154835 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4a0a3f9d-e190-4e25-bc4f-cc828c43b51f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:33:54.610717 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-038366ea-38c1-4f82-8014-e5a8e93c17ac None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:34:03.152147 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-148924ca-b01f-4b30-9d22-661ee94e05d6 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:34:54.615723 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-dc07ba86-d3e6-4429-8d74-15db0c53b759 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:35:03.159047 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-29e376be-c200-44c9-853b-a1d92d057137 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:35:54.612520 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7649b2ab-1ec3-4c57-a464-e8c5a3306357 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:36:03.156681 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a1ad15ec-1f3d-4bce-b17a-058a6e93beec None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:36:48.317034 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Flow 'volume_create_scheduler' (cdbe1aea-f368-4509-a4db-45fcfa183504) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:36:48.319521 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (72e0ab8f-23ca-4844-b9bc-4f4d47838121) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:36:48.321942 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (72e0ab8f-23ca-4844-b9bc-4f4d47838121) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d577fb37-8efe-4a8c-9f30-0b2a4f7f7795),volume_id=d577fb37-8efe-4a8c-9f30-0b2a4f7f7795,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:36:48.323922 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8445d14c-747b-4a8f-bff9-f166c3be8c92) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:36:48.331508 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:36:48.346172 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 36, 3, 152570)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:36:48.346716 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:36:48.347078 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:36:48.347430 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d577fb37-8efe-4a8c-9f30-0b2a4f7f7795) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:36:48.347822 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.16, total_capacity_gb: 19.63,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:36:03.152570 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:36:48.348858 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:36:48.350902 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 361.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:36:48.352885 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:36:48.355260 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:36:48.357772 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:36:48.357772 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:36:48.357772 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:36:48.357772 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.16, total_capacity_gb: 19.63,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:36:03.152570] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:36:48.357772 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Weigher CapacityWeigher returned, weigher value is {max: 381.59999999999997, min: 381.59999999999997} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:36:48.358248 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:36:48.358248 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:36:48.358248 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.16, total_capacity_gb: 19.63,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:36:48.351235 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:36:48.378741 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8445d14c-747b-4a8f-bff9-f166c3be8c92) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:36:48.380859 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-cdbba36d-096f-4728-b433-b340b8bf46ca tempest-ServerActionsTestJSON-1681873499 None] Flow 'volume_create_scheduler' (cdbe1aea-f368-4509-a4db-45fcfa183504) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:36:54.618152 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3fc9f0b0-2342-4166-9547-e6d6bfede378 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:37:03.158973 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7ac5ce53-d098-4d54-ab45-535668460aac None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:37:28.156334 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8eeb9825-db23-4d56-93ff-c05276d2a959 tempest-ServerActionsTestJSON-1681873499 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:37:54.622147 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5f8d5ca2-2686-48b6-b5bb-99e766a1e79f None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:38:03.174178 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f88b8026-1457-409b-af92-b80b5077dd17 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:38:54.628617 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-318886ea-cce0-4d47-9689-9182e48c847c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:39:02.481756 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Flow 'volume_create_scheduler' (15a24954-0169-4cfe-9346-2ae64857fa0f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:39:02.485163 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6015083c-929f-4116-a884-bf7c68de4e2d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:39:02.487924 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6015083c-929f-4116-a884-bf7c68de4e2d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c01d06e1-04b2-415f-8356-72ef323aafd4),volume_id=c01d06e1-04b2-415f-8356-72ef323aafd4,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:39:02.489963 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a4113d4f-05ec-4966-a986-9e880bafd346) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:39:02.498667 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:39:02.513507 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 38, 3, 168995)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:39:02.513920 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:39:02.515091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:39:02.515500 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c01d06e1-04b2-415f-8356-72ef323aafd4) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:39:02.515920 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.16, total_capacity_gb: 19.63,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:38:03.168995 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:39:02.516998 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: if self._filter_one(obj, filter_properties): Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: return passes_method(obj, filter_properties) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: LOG.debug("Space information for volume creation " Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: Arguments: {'available': 383.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: --- Logging error --- Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: Traceback (most recent call last): Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: msg = self.format(record) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: return logging.StreamHandler.format(self, record) + record.reset_color Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: return fmt.format(record) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: return logging.Formatter.format(self, record) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: record.message = record.getMessage() Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: msg = msg % self.args Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: KeyError: 'requested' Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: Call stack: Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: func(*args, **kwargs) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: self.work.run() Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Nov 20 21:39:02.518993 ncdv-master-1 cinder-scheduler[133499]: res = self.dispatcher.dispatch(message) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: return self._do_dispatch(endpoint, method, ctxt, args) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: result = func(ctxt, **new_args) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: return caller(func, *(extras + args), **kw) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: result = f(*args, **kwargs) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: return schedule_function(*args, **kwargs) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: flow_engine.run() Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: for _state in self.run_iter(timeout=timeout): Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: for transition in r.run_iter(builder.START): Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: event = cb(old_state, new_state, event, *args, **kwargs) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: not_done, failures = do_schedule(memory.next_up) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: return self._scheduler.schedule( Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: futures.add(scheduler.schedule(atom)) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: return self._task_action.schedule_execution(task) Nov 20 21:39:02.521352 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: return self._task_executor.execute_task( Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: fut = self._executor.submit(_execute_task, Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: return self._gatherer.submit(fn, *args, **kwargs) Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: fut = self._submit_func(fn, *args, **kwargs) Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: result = self.fn(*self.args, **self.kwargs) Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: result = task.execute(**arguments) Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: self.driver_api.schedule_create_volume(context, request_spec, Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: backend = self._schedule(context, request_spec, filter_properties) Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: weighed_backends = self._get_weighted_candidates(context, request_spec, Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: backends = self.host_manager.get_filtered_backends(backends, Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: return self.filter_handler.get_filtered_objects(filter_classes, Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Nov 20 21:39:02.523452 ncdv-master-1 cinder-scheduler[133499]: list_objs = list(objs) Nov 20 21:39:02.525127 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:39:02.525127 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:39:02.525127 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.16, total_capacity_gb: 19.63,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:38:03.168995] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:39:02.525127 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Weigher CapacityWeigher returned, weigher value is {max: 381.59999999999997, min: 381.59999999999997} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:39:02.525481 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:39:02.525481 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:39:02.525481 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.16, total_capacity_gb: 19.63,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:39:02.520213 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:39:02.547308 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a4113d4f-05ec-4966-a986-9e880bafd346) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:39:02.549345 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-1216c350-ae23-4365-9888-c281c8198a88 tempest-TestMinimumBasicScenario-599005410 None] Flow 'volume_create_scheduler' (15a24954-0169-4cfe-9346-2ae64857fa0f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:39:03.164524 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4d4d4dab-7880-4cb5-8e91-53aa34997ed0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:39:54.633928 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-62cf44d0-120a-49d5-9415-dff69b48d720 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:40:03.167826 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-486af6f2-96ad-4cf3-b405-cea85132bea5 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:40:24.523903 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-40f382ae-7758-4de5-92e9-5c12717b4166 tempest-TestMinimumBasicScenario-599005410 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:40:54.630474 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-291dbf7b-becd-4e4b-952d-cefde86bd459 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:41:03.192982 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3fe5bb59-ad80-42d7-97d2-d07d6cc86e79 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:41:54.638954 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-de145218-a8a3-47c5-abd8-85894997adc5 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:42:03.225410 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4938dd34-b452-4111-a023-566fc1660747 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:42:54.641687 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-861ff116-a4e4-4eec-989e-d01e31761c30 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:43:03.167786 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-aeb8d647-0cf4-400b-86f8-03d2e4d71cc8 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:43:54.647563 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d712f87b-cc74-4f57-860b-39ce8c83fb94 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:44:03.169249 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8a9bffcf-9820-431c-b8d4-b73472ff7d5f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:44:54.642667 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-02f2ac64-7c8a-49da-98b0-52f1a56d80a8 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:45:03.167782 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f5ae70d3-2504-4815-b07e-90ad631e552d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:45:54.648182 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-a4bd92df-8f26-41fe-9aae-137faad67c43 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:46:03.169992 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8fbc1c73-8aba-4e37-b945-180a39f865b3 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:46:54.644743 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-20cf8dfa-29ee-4f28-9499-725e62909c89 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:47:03.168459 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8ae6c68-48ca-4749-b430-975430a828c3 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:47:54.650888 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0a4cf12e-9119-44ef-a359-f8dc69b8e6be None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:48:03.172190 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9c916805-aef4-4e5c-a6af-333c52a3a7e4 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:48:54.647824 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7108d3b4-111e-42bf-b63c-97748d32ed3e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:49:03.178782 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-116b7dd0-2b04-4c74-a817-a4bc9bb74d84 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:49:54.651680 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2497c1b9-cf3f-4f7b-9129-27d11ff0985b None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:50:03.174391 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-50b8484f-4825-42fb-a5c8-91b630f1ee6b None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:50:54.657418 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-31cc077a-fe74-43bc-8960-f1b3131fb6fc None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:51:03.202662 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-738a47bf-5a70-4730-8e64-3a62c39abaa4 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:51:54.663086 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-4e0d464e-a618-400d-b11b-1c817dd36502 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:52:03.173371 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8b3bfd8f-6a1e-4a23-8982-6abfc4d75b1e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:52:35.164742 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Flow 'volume_create_scheduler' (9cd26a12-738e-4a41-9ae6-61fa30480b59) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:52:35.167596 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20ae2dc3-15e1-425f-b5f6-f06c8ce8de37) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:52:35.170081 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20ae2dc3-15e1-425f-b5f6-f06c8ce8de37) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=89603187-210f-4d54-a40f-71dfff8bc059,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9820fef1-336c-4a78-9fb3-205790bf830e),volume_id=9820fef1-336c-4a78-9fb3-205790bf830e,volume_properties=VolumeProperties,volume_type=VolumeType(cc126a8a-4799-426e-bb2f-4fee832a3952))}' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:52:35.172026 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (59bf2046-a299-4370-bed7-b972174e4f25) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Nov 20 21:52:35.179872 ncdv-master-1 cinder-scheduler[133499]: DEBUG oslo_db.sqlalchemy.engines [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133499) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Nov 20 21:52:35.195763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 11, 20, 21, 52, 3, 169178)} {{(pid=133499) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Nov 20 21:52:35.195763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Starting with 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Nov 20 21:52:35.195763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:52:35.195763 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9820fef1-336c-4a78-9fb3-205790bf830e) {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Nov 20 21:52:35.196218 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.15, total_capacity_gb: 19.63,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:52:03.169178 {{(pid=133499) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Nov 20 21:52:35.197162 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Filter CapacityFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:52:35.197546 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_filter [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133499) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Nov 20 21:52:35.197930 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.15, total_capacity_gb: 19.63,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:52:03.169178] {{(pid=133499) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Nov 20 21:52:35.198356 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.base_weight [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Weigher CapacityWeigher returned, weigher value is {max: 381.59999999999997, min: 381.59999999999997} {{(pid=133499) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Nov 20 21:52:35.198705 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133499) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Nov 20 21:52:35.199276 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.filter_scheduler [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133499) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Nov 20 21:52:35.199657 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.15, total_capacity_gb: 19.63,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-11-20 21:52:35.199307 {{(pid=133499) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Nov 20 21:52:35.226374 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (59bf2046-a299-4370-bed7-b972174e4f25) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133499) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Nov 20 21:52:35.228412 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.manager [None req-f8791fd0-4e11-4cd7-94f6-4644e661a532 tempest-TestShelveInstance-1713581458 None] Flow 'volume_create_scheduler' (9cd26a12-738e-4a41-9ae6-61fa30480b59) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133499) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Nov 20 21:52:54.659676 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fa8ea4ca-38de-40c9-93d5-24484be7bd31 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:53:03.174835 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-beb394eb-025b-4fd8-978d-d963578305da None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.63, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:53:54.665256 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-98077615-dd14-408a-94a3-9fe326d33ec3 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:54:03.174698 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-5b96da6e-b7c8-447b-9a7c-535a801eebd0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.06, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:54:54.661860 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c6ad272a-9ee1-464b-8628-38bace0f96a0 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:55:03.173902 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8fdb85db-64fa-4ee8-bfe9-7fa4a6c4c2fb None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.59, 'free_capacity_gb': 19.03, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:55:54.658724 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-f0bf79cd-6ae7-4d1b-bb6e-d54ab3e3b5c1 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:56:03.174681 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-559ae474-fe8c-4f02-8773-06b276f38732 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.57, 'free_capacity_gb': 19.0, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:56:49.450460 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [req-913c9d03-3448-4e04-abb1-f563c58ad68b req-0f7bc75d-9d4a-4937-ac5d-a1b3381e0847 tempest-TestShelveInstance-1713581458 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 18.99, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:56:54.656980 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-48b9e1b5-38df-4319-a319-eec631fd8791 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:57:03.206609 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-510996a2-4d97-4a75-8439-433e5e6aedb8 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:57:54.661296 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-30e56c26-7c02-41bf-8d0c-d6beae42f719 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:58:03.175188 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-0bbad2e2-23c2-4beb-b281-6607974657d2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:58:54.657729 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-217bc2b5-bc7a-4c48-8b0f-d38f9da912dd None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 21:59:03.174469 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-3c9047a4-1994-434a-8bf2-6df5fcdd7cb2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 21:59:54.663545 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-d197e120-6c2c-4dca-8746-7dc97ae590c7 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:00:03.172757 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-c1854210-1767-47b5-80b2-c76aa4ecd678 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:00:54.660177 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-2ed34764-c55e-4d99-9b56-57b7cae517e3 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:01:03.171065 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-51ca2758-f93f-43cd-9ce2-dbd7ed8ed7b9 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:01:54.666669 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-fae53059-eec8-4463-958d-80530193d3a4 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:02:03.211766 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-af04e0fb-37e7-492e-887c-31f784fc04b1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:02:54.673850 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-b325ef39-8cf0-4940-b204-cd4ffe6e4349 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:03:03.169417 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-47ef8c9c-650f-46d1-a13a-d3a67f6c96d7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:03:54.678091 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-aedb1bdc-f3a7-4ea5-bd47-7c8737460c7d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:04:03.170661 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9c0a3e4c-f43a-431d-a52a-7d1921ea4b24 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:04:54.682922 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8e960daa-e095-4e5f-99ad-4e29be49765d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:05:03.170845 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7ba4b2ae-4ac3-4bbe-98bb-f78602667867 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:05:54.688006 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-635b8d60-0deb-4590-9572-5d2b7063f6a0 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:06:03.170317 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-7ad4089a-cbe0-4765-aaa1-b1fdec191474 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:06:54.693119 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-9c95b5bf-79db-4fb7-bb84-6ca77f2b07a7 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:07:03.169302 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-82400590-8665-4d77-8306-7e8bdf7d46b7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:07:54.698520 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-e79360cd-bdea-4c00-8154-ed6653524e0b None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:08:03.174938 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cf1e479c-9242-4e3d-bf0b-c579ff10df7d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:08:54.703872 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-baeda832-76c9-4997-bc59-5a204c6fe0e9 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:09:03.166041 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-cdc9cad5-5b42-4fcb-81ce-65196ce1e308 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Nov 20 22:09:54.709825 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-86cdeee5-a119-4e6d-8d14-5807f4faaf53 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Nov 20 22:10:03.167819 ncdv-master-1 cinder-scheduler[133499]: DEBUG cinder.scheduler.host_manager [None req-8aa2c07f-e7e9-463d-a0e6-c5ebce177dc1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.56, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:5cc0e352-4d7c-4213-841d-37245735ca3f:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133499) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}}